Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable CLI JSON output for script consumption #746

Merged
merged 1 commit into from
Jul 10, 2020

Conversation

kevin-bates
Copy link
Member

This change allows for elyra-metadata list <namespace> --json output to be piped to JSON-aware tools for consumption. It allows for usage w/ and w/o the --valid-only flag where invalid entries will have a reason with a value entry.

Errors still appear on stderr but stdout can be piped or redirected.

Resolves #742

Here are some outputs after piping to jq...

With --valid-only...

$ elyra-metadata list runtimes --json --valid-only |jq
[E 2020-07-09 17:03:36,372.372] JSON failed to load for metadata 'foo' in namespace 'runtimes' with error: Expecting ',' delimiter: line 10 column 5 (char 318).
[E 2020-07-09 17:03:36,377.377] Schema validation failed for metadata 'foofoo' in namespace 'runtimes' with error: None is not of type 'string'.
[E 2020-07-09 17:03:36,379.379] Schema validation failed for metadata 'barbar' in namespace 'runtimes' with error: 'xxx' is too short.
[
  {
    "name": "foodoo",
    "display_name": "Foo Doo",
    "metadata": {
      "api_endpoint": "http://wackwach",
      "cos_endpoint": "http://zackzach",
      "cos_username": "cos_username",
      "cos_password": "cos_secret",
      "cos_bucket": "cos-chuckit"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/foodoo.json"
  },
  {
    "name": "baz",
    "display_name": "BAZ",
    "metadata": {
      "api_endpoint": "http://codait.org",
      "cos_endpoint": "http://baz.boo",
      "cos_username": "charles",
      "cos_password": "farles64",
      "cos_bucket": "chuckit"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/baz.json"
  },
  {
    "name": "kf_1",
    "display_name": "KubeFlow 1.0",
    "metadata": {
      "api_endpoint": "http://cloning1.fyre.ibm.com:31380/pipeline",
      "cos_endpoint": "http://cloning1.fyre.ibm.com:31671",
      "cos_username": "minio",
      "cos_password": "minio123",
      "cos_bucket": "kbates"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/kf_1.json"
  }
]

Without --valid-only so invalid entries are included...

$ elyra-metadata list runtimes --json |jq
[E 2020-07-09 17:04:35,389.389] JSON failed to load for metadata 'foo' in namespace 'runtimes' with error: Expecting ',' delimiter: line 10 column 5 (char 318).
[E 2020-07-09 17:04:35,394.394] Schema validation failed for metadata 'foofoo' in namespace 'runtimes' with error: None is not of type 'string'.
[E 2020-07-09 17:04:35,395.395] Schema validation failed for metadata 'barbar' in namespace 'runtimes' with error: 'xxx' is too short.
[
  {
    "name": "foodoo",
    "display_name": "Foo Doo",
    "metadata": {
      "api_endpoint": "http://wackwach",
      "cos_endpoint": "http://zackzach",
      "cos_username": "cos_username",
      "cos_password": "cos_secret",
      "cos_bucket": "cos-chuckit"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/foodoo.json"
  },
  {
    "name": "baz",
    "display_name": "BAZ",
    "metadata": {
      "api_endpoint": "http://codait.org",
      "cos_endpoint": "http://baz.boo",
      "cos_username": "charles",
      "cos_password": "farles64",
      "cos_bucket": "chuckit"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/baz.json"
  },
  {
    "name": "kf_1",
    "display_name": "KubeFlow 1.0",
    "metadata": {
      "api_endpoint": "http://cloning1.fyre.ibm.com:31380/pipeline",
      "cos_endpoint": "http://cloning1.fyre.ibm.com:31671",
      "cos_username": "minio",
      "cos_password": "minio123",
      "cos_bucket": "kbates"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/kf_1.json"
  },
  {
    "name": "foofoo",
    "display_name": "Foo Baz",
    "metadata": {
      "description": null,
      "api_endpoint": "http://wackwach",
      "cos_endpoint": "http://zackzach",
      "cos_username": "cos_username",
      "cos_password": "cos_password",
      "cos_bucket": "cos_chuckit"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/foofoo.json",
    "reason": "ValidationError"
  },
  {
    "name": "barbar",
    "display_name": "KFP with IBM COS (yukked1)",
    "metadata": {
      "api_endpoint": "http://yukked1.fyre.ibm.com:31380/pipeline",
      "cos_endpoint": "https://s3.us-south.cloud-object-storage.appdomain.cloud",
      "cos_username": "xxx",
      "cos_password": "xxx",
      "cod_secure": "True",
      "cos_bucket": "xxx"
    },
    "schema_name": "kfp",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/barbar.json",
    "reason": "ValidationError"
  },
  {
    "name": "foo",
    "display_name": null,
    "metadata": {},
    "schema_name": "{unknown}",
    "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/foo.json",
    "reason": "ValueError"
  }
]

And when no pipe/redirection is used...

$ elyra-metadata list runtimes --json --valid-only 
[E 2020-07-09 17:04:12,716.716] JSON failed to load for metadata 'foo' in namespace 'runtimes' with error: Expecting ',' delimiter: line 10 column 5 (char 318).
[E 2020-07-09 17:04:12,722.722] Schema validation failed for metadata 'foofoo' in namespace 'runtimes' with error: None is not of type 'string'.
[E 2020-07-09 17:04:12,723.723] Schema validation failed for metadata 'barbar' in namespace 'runtimes' with error: 'xxx' is too short.
[{
  "name": "foodoo",
  "display_name": "Foo Doo",
  "metadata": {
    "api_endpoint": "http://wackwach",
    "cos_endpoint": "http://zackzach",
    "cos_username": "cos_username",
    "cos_password": "cos_secret",
    "cos_bucket": "cos-chuckit"
  },
  "schema_name": "kfp",
  "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/foodoo.json"
}, {
  "name": "baz",
  "display_name": "BAZ",
  "metadata": {
    "api_endpoint": "http://codait.org",
    "cos_endpoint": "http://baz.boo",
    "cos_username": "charles",
    "cos_password": "farles64",
    "cos_bucket": "chuckit"
  },
  "schema_name": "kfp",
  "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/baz.json"
}, {
  "name": "kf_1",
  "display_name": "KubeFlow 1.0",
  "metadata": {
    "api_endpoint": "http://cloning1.fyre.ibm.com:31380/pipeline",
    "cos_endpoint": "http://cloning1.fyre.ibm.com:31671",
    "cos_username": "minio",
    "cos_password": "minio123",
    "cos_bucket": "kbates"
  },
  "schema_name": "kfp",
  "resource": "/Users/kbates/Library/Jupyter/metadata/runtimes/kf_1.json"
}]

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

This change allows for `elyra-metadata list <namespace> --json` output to
be piped to JSON-aware tools for consumption.  It allows for usage w/ and
w/o the `--valid-only` flag where invalid entries will have a `reason`
with a value entry.

Errors still appear on stderr (as expected) but stdout can be piped or
redirected.
@lresende lresende merged commit a76c5f5 into elyra-ai:master Jul 10, 2020
@kevin-bates kevin-bates deleted the enable-cli-json-for-scripting branch August 3, 2020 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

metadata cli: support JSON-formatted output
2 participants