/
Marquez - Community Meetings & Calendar

Marquez - Community Meetings & Calendar

Mar 20, 2024

 

Marquez Monthly Community Meeting

The Marquez Community Meeting occurs on the fourth Thursday of each month. Meetings are held on Zoom.

Next meeting: March 28, 2024

Tentative agenda:

  1. Announcements

  2. New WIP Tagging Feature: data governance and other relevant use cases [Willy Lulciuc]

  3. New WIP Tagging Feature: UI implementation [Peter Hicks]

  4. open discussion

February 22, 2024

Agenda:

  1. announcements

  2. discussion: docs and landing pages updates

  3. discussion: learnings about performance issues and possible solutions

  4. open discussion

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Michael Robinson, Community Team, Astronomer

    • Julien Le Dem, Project Lead, OpenLineage

  • And:

    • Harsh Loomba, Upgrade

    • David Goss, Staff Software Engineer, Matillion

    • David Sharp, ANZ

January 25, 2024

Tentative agenda:

  1. announcements

  2. recent release

  3. 2024 Roadmap

  4. demos of UI tag support for datasets and a NEW UI redesign for operational lineage and column-level lineage

December 7, 2023

Tentative agenda:

  1. announcements

  2. 2023 recap

  3. 2024 roadmap discussion

  4. open discussion

October 26, 2023

Tentative agenda:

  1. announcements

  2. recent releases

  3. I/O tab addition

  4. hover-over tooltip for dataset tags demo by @David Sharp

  5. static lineage progress update

  6. open discussion

September 28, 2023

Agenda:

  1. Announcements

  2. Recent releases

  3. Recent API changes

  4. Recent Web UI changes

  5. Static lineage and streaming support update

  6. Discussion

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Michael Robinson, Community Team, Astronomer

    • Julien Le Dem, Project Lead, OpenLineage

    • John Lukenoff, Software Engineer, Asana

  • And:

    • Harel Shein, Director of Engineering, Astronomer

    • Paweł Leszczyński, Data Engineer, GetInData

    • David Goss, Staff Software Engineer, Matillion

    • David Sharp, ANZ

August 24, 2023

Agenda:

  1. Announcements

  2. Recent releases

  3. Applying data retention for Marquez at Astronomer

  4. Static lineage support update

  5. Open discussion

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Michael Robinson, Community Team, Astronomer

    • Julien Le Dem, Project Lead, OpenLineage

  • And:

    • David Goss, Matillion

    • Harsh Loomba, Upgrade

    • Harel Shein, Engineering Director, Astronomer

Notes:

  • Announcements [Willy]

    • Upcoming event: Airflow Summit, September 19-21 in Toronto

      • Airflow talks and OpenLineage meetup

      • Julien and Willy will be attending

    • Upcoming meetup: the first Marquez meetup will be happening in San Francisco in October

    • LF AI Graduation presentation coming up on September 7th

  • Recent releases [Michael R.]

    • 0.38.0

    • 0.39.0

    • 0.40.0

  • Applying data retention for Marquez at Astronomer [Willy]

  • Static lineage support update [Willy]

  • Open discussion

July 27, 2023

Agenda:

  1. Announcements

  2. Recent releases

  3. Applying data retention for Marquez at Astronomer

  4. Open discussion

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Michael Robinson, Community Team, Astronomer

June 22, 2023

Agenda:

  1. Announcements

  2. Recent releases

  3. Lineage graph cycling fix in 0.35.0

  4. Datasets pagination in 0.35.0

  5. Open discussion

Meeting:

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Michael Robinson, Community Team, Astronomer

    • John Lukenoff, Software Engineer, Asana

May 25, 2023

Agenda:

  1. Announcements

  2. Updates

  3. Recent releases

  4. Db retention change

  5. inputFacets and outputFacets change

  6. Discussion items

    1. compaction to fix unnecessary duplication

  7. Open discussion

April 27, 2023

Agenda:

  1. Roadmap discussion

  2. Open discussion

Meeting:

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Michael Robinson, Community Team, Astronomer

  • And:

    • Harel Shein, Director of Engineering, Astronomer

Notes:

  • Roadmap and OpenLineage v2 spec discussion [Willy]

    • Feedback on OpenLineage 2 discussions will be important because the evolution of the spec will require work on Marquez that will need to be aligned with the release

    • One issue in the discussions: whether to introduce a breaking change in OpenLineage, or support static lineage in a backwards-compatible way

      • Why would we want to be backwards-compatible?

        • It's still a young spec, but there have been deep integrations with Airflow and others. We've talked to stakeholders who have said they will stay on version 1 for a while rather than migrate in the case of a breaking change.

      • What additional work will we need to do on the backend to make static lineage work?

        • Marquez has always been focused on the job run specifically: it extracts inputs and outputs as events are coming in. The logic includes creating a job version when this happens. We version the output dataset. If we don't have that run anymore, what will we need to change?

        • The job and entity represented in the run event would no longer be there, so you would not have a job name, but maybe there would be facets associated with the job itself.

        • Example: Snowflake. They don't necessarily have a job, but they have query logs. These are an example of when something runs in the background is a job but it's just there to facilitate the query.

          • Query log: good way to say there was a run but there are no job details. But we do know the inputs/outputs and the tables, giving us more flexibility to capture lineage details for cases where jobs don't exist.

          • Do we implement dummy job names?

        • Marquez has been very opinionated with APIs in the model itself. Is it now loosening up in order to adapt to the changing OpenLineage standard?

          • I think it's a natural expansion of the model where you don't always have the discrete entities running SQL or code. A lot of these things can be ephemeral, which we're trying to address with the changes to the spec – one-time instances, etc.

      • Peter, you mentioned at the meetup last night that this spec change could have some implications for the UI and the user experience.

        • Peter: it's a bit of a challenge because what will, I think, be generated out of this will be a lot of these very small graphs. You have an entity, whatever. We're going to call that a jobless entity and its output. But you'll be generating a lot of that kind of small graph, and I think the UI is not really constructed around that case. So we'll have to probably think about how that exists, how we can maybe aggregate the same jobless entities together and have some heuristic based on that.

        • Facets might be optional.

        • Maybe we just base it off the run information we get – database logs where we have some timestamps available to us. The best example we have is the Snowflake adapter that's querying historical query logs. We could funnel that information into Marquez and see what we could do to generate a job name that would be consistent across different cases where job information doesn't exist.

      • There would be work needed to support new types that would make events unserializable despite being valid OpenLineage events. 

        • Using the OpenLineage server models, which are more flexible, while keeping the initial high-level fields, would be one option.

          • Marquez always had a path to create a dataset (one of the proposed new types)

          • We're coming full-circle because Marquez had static lineage support at one time. We had registered datasets for jobs, then behind the scenes Marquez would create a default new state for a job that hadn't run yet.

          • At the time, the feedback was that it is a lot of work to maintain all these API calls, and it would be nice if everything was provided in the payload.

          • OpenLineage came out, and we defined everything that's specific to a run, including the job and inputs and outputs, and made a simple call.

          • Now what we're hearing from the community is that in some cases I don't have a run and I just want to register static lineage.

      • How would you see the migration working?

        • The endpoint would handle any of the types. There would be different types, but this wouldn't change the run. Everything remains the same through the contract. It's just you have additional event types, which naturally happens when you see more usage and use cases in the wild.

        • The API might need to do a little split. Static lineage will be quicker to retrieve.

      • What is the expected timeline?

        • There's no timeline at the moment but for sure this year or this quarter, or next quarter.

March 23, 2023

Agenda:

  1. Announcements

  2. Recent releases

  3. A recent jobs symlinks fix

  4. Discussion items:

    1. BI support

    2. Implications of design lineage support in the OL spec

  5. Open discussion

Meeting:

Slides:

Attendance:

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Julien Le Dem, Chief Architect, Astronomer

  • And:

    • Michael Robinson, Software Engineer, Dev. Rel., Astronomer

    • Minkyu Park, Senior Engineer, Astronomer

    • John Thomas, Software Engineer, Dev. Rel., Astronomer

    • Benji Lampel, Product Manager, Astronomer

    • Bruno Cavestro, BI engineer curious about potentiality of the tool

Notes:

  • Announcements [Michael R.]

    • Mar. 30: Julien (@julienledem) speaking at Data Council Austin

    • Mar. 30: OpenLineage Meetup at Data Council Austin

    • Mar. 28-30: Ross (@rossturk), Pawel (@pawel-big-lebowski) and Maciej (@mobuchowski) speaking at Big Data Tech Warsaw 2023

  • Recent Release 0.32.0 [Michael R.]

  • Discussion Items

    • BI Support [Bruno]

      • What does BI mean in connection to Marquez?

      • BI dashboard not yet part of the model, but static lineage might be relevant to BI [Julien]

      • Key metrics often part of BI, and these must be connected to lineage; a BI facet has been discussed, but introducing it presents a challenge [Willy]

      • How do machine learning models come into play? These might not be traditional BI tools but are avenues to explore. [Benji]

      • Basic, ideal situation: aggregation and documentation of where KPI is used in a chart, etc.

      • Problems include: BI tools generate internal data models or BI dashboards touch a company's main production database, making integrating OpenLineage difficult

    • RunEvent-less Metadata Support in OpenLineage [Willy]

      • Motivation: make community aware of RunEvent-less metadata emission proposal

      • We need to define a "v 2" of OpenLineage to define events outside of the run context. How would Marquez define lineage outside a run?

      • The run is critical to versioning in Marquez

      • Static lineage would mean focusing on the relationship between job and dataset, rather than also on the job and run

        • Julien: IMO, the only thing excluded from the current model would be the run

          • The dataset <> job version layer would be updated

        • Mike: we could uncouple the relationship between inputs/outputs and job versions

          • I would love to see a CI/CD integration

          • The only version info it would need would be the version of the code

          • Versioning info could also come from integrations, e.g., Iceberg, and be supplemented as needed.

        • Willy: another approach: implicit and explicit versioning

      • REST API > lineage event reprioritization

      • Do we need to define new events, e.g. a dataset event?

      • What are the implications for Marquez of events without run IDs?

      • Proposed model:

        • JSON schema would persist for datasets and jobs

        • inputs and outputs would be predefined

      • Recommended: review the document and give input on where the standard should go.

      • Timeline for feedback is not firm, but the discussion is accelerating.

February 23, 2023

Attendees

  • TSC:

    • Willy Lulciuc, Co-creator of Marquez

    • Peter Hicks, Senior Engineer, Astronomer

    • Julien Le Dem, Chief Architect, Astronomer

  • And:

    • Michael Robinson, Software Engineer, Developer Relations, Astronomer

    • Minkyu Park, Senior Engineer, Astronomer

    • John Thomas, Software Engineer, Dev. Rel., Astronomer

    • Prachi Mishra, Senior Engineer, Astronomer

    • Ross Turk, Senior Director of Community, Astronomer

    • Benji Lampel, Product Manager, Astronomer

    • Bruno Cavestro, BI engineer curious about potentiality of the tool

Agenda

  • Announcements

  • Recent releases

  • LFAI progress update

  • UI improvements demo

  • Open discussion

Meeting

Slides:

Notes