Tech
Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy
A Spark job commits a table update. The catalog writes the change to Postgres. Then the network drops between the catalog and the client, and the client never sees the response. The client does the sensible thing and retries. This time the catalog sees that the table has already moved past the base snapshot in the request, so it returns 409 Conflict. The client reads that 409 as a failed commit and deletes the metadata files it just wrote. The commit is now recorded in the catalog, and the files...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to