Skip to content

feat: Pass extra configuration to json.dump() - #622

Closed
Rutam21 wants to merge 2 commits into
apify:masterfrom
Rutam21:IS-526
Closed

Rutam21 wants to merge 2 commits into
apify:masterfrom
Rutam21:IS-526

Conversation

@Rutam21

@Rutam21 Rutam21 commented Oct 25, 2024

Copy link
Copy Markdown
Contributor

Description

This PR addresses the need for customizable export configurations when using the await crawler.export_data("export.json") method.

Key Changes:

  • Introduced a new option in the export_data method to accept additional keyword arguments, allowing users to pass configurations directly to json.dump() and other export functions.
  • Created dedicated export helper methods for different formats in the BasicCrawler, including export_data_json and export_data_csv. This separation enhances clarity and maintainability while providing format-specific options.

This enhancement improves the flexibility of data exports, enabling users to tailor their export settings to meet specific requirements.

Issues

Fixes #526

Testing

Unit test cases are added to check the updated configuration and functionality.

Checklist

  • CI passed

@Rutam21

Rutam21 commented Oct 28, 2024

Copy link
Copy Markdown
Contributor Author

This PR is based on all the discussions in the parent issue and the suggestions thereafter. Please review and suggest changes, if any. Thank you.

cc: @vdusek @janbuchar

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure the CI passes, see the CONTRIBUTING.md for more information.

@janbuchar
janbuchar self-requested a review October 31, 2024 10:29

@janbuchar janbuchar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI is still failing, can you please fix the issues?

@Rutam21

Rutam21 commented Oct 31, 2024

Copy link
Copy Markdown
Contributor Author

@janbuchar I will push a commit to fix some of them by EOD.

@janbuchar

Copy link
Copy Markdown
Collaborator

@Rutam21 thank you for your initiative, but we decided to just fix up #580 on our own and merge that.

@janbuchar janbuchar closed this Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement/document a way how to pass extra configuration to json.dump()

3 participants