CHANGELOG.md 15 KB
Change Log
v0.53
- #332:
- Switch to Pyarrow's S3FileSystem to read from S3 on the Forget Phase
- Switch to boto3's upload_fileobj to write to S3 on the Forget Phase
- Upgrade backend dependencies
- #327: Improve performance of Athena results handler
- #329: Upgrade frontend dependencies
v0.52
v0.51
- #321: Upgrade numpy dependency
v0.50
- #322: Upgraded to Python 3.9
v0.49
- #314: Fix query generation step for Composite matches consisting of a single column
- #320: Fix deployment issue introduced in v0.48
v0.48
- #316: Upgrade dependencies
- #313: Add option to choose IAM for authentication (in place of Cognito)
- #313: Add option to not deploy WebUI component. Cognito auth is required for WebUI
v0.47
- #310: Improve performance of Athena query generation
- #308: Upgrade frontend dependencies and use npm workspaces to link frontend sub-project
v0.46
- #306: Adds retry behaviour for old object deletion to improve reliability against transient errors from Amazon S3
v0.45
- #303: Improve performance of Athena query generation
- #301: Include table name to error when query generation fails due to an invalid column type
- Dependency version updates for:
v0.44
- #293: Upgrade dependencies
v0.43
#289: Upgrade frontend dependencies
#287: Add data mapper parameter for ignoring Object Not Found exceptions encountered during deletion
v0.42
#285: Fix for a bug that caused a job to fail with a false positive
The object s3://<REDACTED> was processed successfully but no rows required deletionwhen processing a job with queries running for more than 30m#286: Fix for a bug that causes
AthenaQueryMaxRetriessetting to be ignored#286: Make state machine more resilient to transient failures by adding retry
#284: Improve performance of find query for data mappers with multiple column identifiers
v0.41
#283: Fix for a bug that caused a job to fail with
Runtime.ExitErrorwhen processing a large queue of objects to be modified#281: Improve performance of query generation step for tables with many partitions
v0.40
- #280: Improve performance for large queues of composite matches
v0.39
- #279: Improve performance for large queues of simple matches and logging additions
v0.38
- #278: Fix for a bug that caused a job to fail if the processing of an object took longer than the lifetime of its IAM temporary access credentials
v0.37
#276: First attempt for fixing a bug that causes the access token to expire and cause a Job to fail if processing of an object takes more than an hour
#275: Upgrade JavaScript dependencies
#274: Fix for a bug that causes deletion to fail in parquet files when a data mapper has multiple column identifiers
v0.36
- #272: Introduce a retry mechanism when running Athena queries
v0.35
- #271: Support for decimal type column identifiers in Parquet files
v0.34
- #270: Fix for a bug affecting the front-end causing a 403 error when making a request to STS in the Data Mappers Page
v0.33
#266: Fix creating data mapper bug when glue table doesn't have partition keys
#264: Upgrade frontend dependencies
#263: Improve bucket policies
#261: Upgrade frontend dependencies
v0.32
- #260: Add Stockholm region
v0.31
v0.30
- #257: Introduce data mapper setting to specify the partition keys to be used when querying the data during the Find Phase
v0.29
- #256: Upgrade backend dependencies
v0.28
- #252: Upgrade frontend and backend dependencies
v0.27
- #248: Fix for a bug affecting Deletion Jobs running for cross-account buckets
- #246: Upgrade build dependencies
v0.26
v0.25
This version introduces breaking changes to the API and Web UI. Please consult the migrating from <=v0.24 to v0.25 guide
- #239: Remove limit on queue size for individual jobs.
v0.24
- #240: Add ECR API Endpoint and migrate to Fargate Platform version 1.4.0
v0.23
- #238: Upgrade frontend dependencies
v0.22
- #236: Export API Gateway URL + Deletion Queue Table Stream ARN from main CloudFormation Template
v0.21
- #232: Fix for a bug affecting the Frontend not rendering the Data Mappers list when a Glue Table associated to a Data Mapper gets deleted
- #233: Add GET endpoint for specific data mapper
- #234: Performance improvements for the query generation phase
v0.20
v0.19
v0.18
- #223: This release fixes an issue (#222) where new deployments of the solution could fail due to unavailability of a third-party dependency. Container base images are now retrieved and bundled with each release.
v0.17
- #220: Fix for
a bug affecting Parquet files with lower-cased column identifiers generating a
Apache Arrow processing error: 'Field "customerid" does not exist in table schemaexception during the Forget phase (for examplecustomerIdin parquet file being mapped to lower-casecustomeridin glue table)
v0.16
- #216: Fix for
a bug affecting Parquet files with complex data types as column identifier
generating a
Apache Arrow processing error: Mix of struct and list types not yet supportedexception during the Forget phase - #216: Fix for
a bug affecting workgroups other than
primarygenerating a permission error exception during the Find phase
v0.15
- #215: Support for data registered with AWS Lake Formation
v0.14
- #213: Fix for a bug causing a FIND_FAILED error related to a States.DataLimitExceed exception triggered by Step Function's Athena workflow when executing the SubmitQueryResults lambda
- #208: Fix bug preventing PUT DataMapper to edit existing datamapper with same location, fix Front-end DataMapper creation to prevent editing an existing one.
v0.13
- #207: Upgrade frontend dependencies
v0.12
- #202: Fix a
bug that was affecting Partitions with non-string types generating a
SYNTAX_ERROR: line x:y: '=' cannot be applied to integer, varchar(z)exception during the Find Phase - #203: Upgrade frontend dependencies
- #204: Improve performance during Cleanup Phase
- #205: Fix a UI issue affecting FireFox preventing to show the correct queue size due to a missing CORS header
v0.11
- #200: Add API Endpoint for adding deletion queue items in batch - deprecates PATCH /v1/queue
- #170: JSON support
v0.10
- #193: Add support for datasets with Pandas indexes. Pandas indexes will be preserved if present.
- #194: Remove debugging code from Fargate task
- #195: Fix support for requester pays buckets
- #196: Upgrade backend dependencies
- #197: Fix duplicated query executions during Find Phase
v0.9
This version introduces breaking changes to the CloudFormation templates. Please consult the migrating from <=v0.8 to v0.9 guide
v0.8
- #185: Fix dead links to VPC info in docs
- #186: Fix: Solves an issue where the forget phase container could crash when redacting numeric Match IDs from its logs
- #187: Dependency version updates for react-scripts
v0.7
- #183: Dependency version updates for elliptic
v0.6
- #173: Show column types and hierarchy in the front-end during Data Mapper creation
- #173: Add support for char, smallint, tinyint, double, float
- #174: Add support for types nested in struct
- #177: Reformat of Python source code (non-functional change)
- Dependency version updates for:
v0.5
- #172: Fix for an issue where Make may not install the required Lambda layer dependencies, resulting in unusable builds.
v0.4
- #171: Fix for a bug affecting the API for 5xx responses not returning the appropriate CORS headers
v0.3
- #164: Fix for a bug affecting v0.2 deployment via CloudFormation
v0.2
- #161: Fix for
a bug affecting Parquet files with nullable values generating a
Table schema does not match schema used to create fileexception during the Forget phase
v0.1
Initial Release