Upload newman runs to Postman #2849

UmedJadhav · 2021-09-27T10:41:10Z

Adding the ability to upload newman run results to Postman .

Things done

Added a utility to upload newman runs using async-retry
Introduce 4 new CLI parameters
- --publish-workspace <workspace-id>
- --publish-workspace-skip-response
- --publish-retry
- --publish-upload-timeout
Add tests for upload run functionality
Add code for Run Object Serialiser

To-Do List

Add tests for Run Object Serialiser
Add tests for ingesting the new introduced CLI parameters

Refer to the following image for data flow

- This utility will be used to upload the newman runs to postman - We will be using exponential retry so as to not bombard the postman servers

codecov · 2021-09-27T10:42:01Z

Codecov Report

Merging #2849 (7a1b370) into develop (18e0b53) will decrease coverage by 77.96%.
The diff coverage is 7.37%.

@@             Coverage Diff              @@
##           develop    #2849       +/-   ##
============================================
- Coverage    90.96%   13.00%   -77.97%     
============================================
  Files           21       22        +1     
  Lines         1151     1269      +118     
  Branches       349      383       +34     
============================================
- Hits          1047      165      -882     
- Misses         104     1104     +1000

Flag	Coverage Δ
cli	`13.00% <7.37%> (-69.11%)`	⬇️
integration	`11.89% <7.37%> (-29.11%)`	⬇️
library	`11.89% <7.37%> (-46.66%)`	⬇️
unit	`11.89% <7.37%> (-63.17%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
bin/newman.js	`14.28% <0.00%> (-75.52%)`	⬇️
lib/util.js	`19.41% <0.00%> (-75.83%)`	⬇️
lib/run/upload-run.js	`8.04% <8.04%> (ø)`
lib/run/index.js	`9.28% <14.28%> (-84.47%)`	⬇️
bin/util.js	`0.00% <0.00%> (-100.00%)`	⬇️
lib/reporters/json/index.js	`0.00% <0.00%> (-100.00%)`	⬇️
lib/reporters/junit/index.js	`0.00% <0.00%> (-94.05%)`	⬇️
lib/reporters/emojitrain.js	`0.00% <0.00%> (-93.34%)`	⬇️
lib/reporters/progress.js	`0.00% <0.00%> (-92.86%)`	⬇️
lib/run/summary.js	`6.12% <0.00%> (-91.84%)`	⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 18e0b53...7a1b370. Read the comment docs.

coditva · 2021-09-29T05:47:23Z

bin/newman.js

@@ -61,9 +61,12 @@ program
    .option('--cookie-jar <path>', 'Specify the path to a custom cookie jar (serialized tough-cookie JSON) ')
    .option('--export-cookie-jar <path>', 'Exports the cookie jar to a file after completing the run')
    .option('--verbose', 'Show detailed information of collection run and each request sent')
+    .option('-p, --publish-workspace <workspace-id>', 'Publishes to given workspace')
+    .option('--publish-workspace-skip-response', 'Skip responses (headers, body, etc) while uploading newman run to Postman')
+    .option('--publish-retry', 'Number of times newman can try to publish before safely erroring out.')


--publish-retries would make more sense, I think. Thoughts?

Makes sense

coditva · 2021-09-29T05:48:08Z

bin/newman.js

@@ -61,9 +61,12 @@ program
    .option('--cookie-jar <path>', 'Specify the path to a custom cookie jar (serialized tough-cookie JSON) ')
    .option('--export-cookie-jar <path>', 'Exports the cookie jar to a file after completing the run')
    .option('--verbose', 'Show detailed information of collection run and each request sent')
+    .option('-p, --publish-workspace <workspace-id>', 'Publishes to given workspace')
+    .option('--publish-workspace-skip-response', 'Skip responses (headers, body, etc) while uploading newman run to Postman')


What does "workspace" mean in this option? Could it simply be --publish-skip-response?

Changed all the CLI parameter names

coditva · 2021-09-29T05:51:34Z

bin/newman.js

@@ -61,9 +61,12 @@ program
    .option('--cookie-jar <path>', 'Specify the path to a custom cookie jar (serialized tough-cookie JSON) ')
    .option('--export-cookie-jar <path>', 'Exports the cookie jar to a file after completing the run')
    .option('--verbose', 'Show detailed information of collection run and each request sent')
+    .option('-p, --publish-workspace <workspace-id>', 'Publishes to given workspace')


Could it simply be --publish? From the perspective of user, I feel if we have a option like --publish-workspace, we would have another option --publish which enables publishing. But this option itself is doing that, no?

Also, publish-workspace feels a little like we're publishing a workspace. 😅

Thoughts?

cc @codenirvana

I agree, let's keep it --publish

and --publish-skip-response

coditva · 2021-09-29T05:52:05Z

bin/newman.js

+    .option('-p, --publish-workspace <workspace-id>', 'Publishes to given workspace')
+    .option('--publish-workspace-skip-response', 'Skip responses (headers, body, etc) while uploading newman run to Postman')
+    .option('--publish-retry', 'Number of times newman can try to publish before safely erroring out.')
+    .option('--publish-upload-timeout', 'Timeout for uploading newman runs to postman', util.cast.integer, 0)


Could simply be --publish-timeout, I think.

coditva · 2021-09-29T05:53:58Z

bin/newman.js

            const runError = err || summary.run.error || summary.run.failures.length;

            if (err) {
                console.error(`error: ${err.message || err}\n`);
                err.friendly && console.error(`  ${err.friendly}\n`);
            }
-            runError && !_.get(options, 'suppressExitCode') && process.exit(1);
+
+            runError && !newmanStatus.resultUploaded && !_.get(options, 'suppressExitCode') && process.exit(1);


This seems incorrect. If the runError is true and resultUploaded is also true, it would not exit the process with status 1, which it should given that the run has errored.

This should be an ||.

Suggested change

runError && !newmanStatus.resultUploaded && !_.get(options, 'suppressExitCode') && process.exit(1);

(runError || !newmanStatus.resultUploaded) && !_.get(options, 'suppressExitCode') && process.exit(1);

coditva · 2021-09-29T07:43:31Z

lib/run/upload-run.js

+     * @returns {Promise}
+     */
+    _uploadWithRetry = (upload, retryOptions) => {
+        return retry( upload,{


Weird spacing.

Suggested change

return retry( upload,{

return retry(upload, {

coditva · 2021-09-29T07:47:41Z

lib/run/upload-run.js

+     */
+    start = async () => {
+        if(!this.uploadConfig.publishWorkspace){
+            return true;


Shouldn't this check be made before calling the start method itself? 🤔

Also, since this is an async function. It would be better to reject the returned promise by throw-ing here (and everywhere you want to return false) with appropriate error message.

Moved the check to run/index.js

coditva · 2021-09-29T07:48:55Z

lib/run/upload-run.js

+                    Visit ${response.message} to view the results in postman web`);
+            return true;
+
+        } catch(error) {


Let the caller catch the error and show appropriate message.

Reworked the flow

coditva · 2021-09-29T07:51:37Z

lib/run/upload-run.js

+
+        const uploadOptions = {
+            url: UPLOAD_RUN_API_URL,
+            body: JSON.stringify(run),


JSON.stringify will throw in some cases. Verify that those cases won't ever occur or catch the error to handle those cases.

Added a try catch block

coditva · 2021-09-29T07:52:54Z

lib/run/upload-run.js

+            body: JSON.stringify(run),
+            headers: {
+                'Content-Type': 'application/json',
+                'X-API-Header': uploadConfig.postmanApiKey


Is this the correct key for key header? Shouldn't it be X-Api-Key? 🤔

Added Run Serialiser

coditva · 2021-10-06T12:30:34Z

lib/run/index.js

+
+                    // TODO: Add analytics call here
+
+                    if (err) { // TODO -> understand when this is raised


@UmedJadhav This error is raised from the postman-runtime library if the collection run did not finish successfully.

coditva · 2021-10-06T12:32:35Z

lib/run/upload-run.js

+ */
+_upload = (uploadOptions) => {
+    return async (bail) => request.post(uploadOptions , (error, response, body) => {
+                if(error){


Indentation looks weird here.

coditva · 2021-10-06T12:33:16Z

lib/run/upload-run.js

+_upload = (uploadOptions) => {
+    return async (bail) => request.post(uploadOptions , (error, response, body) => {
+                if(error){
+                    if(error.code === 'ECONNREFUSED') { // Retry only if the ERROR is ERRCONNECT


Nitpick: 'ECONNREFUSED' should be a const.

coditva · 2021-10-06T12:34:54Z

lib/run/upload-run.js

+    return async (bail) => request.post(uploadOptions , (error, response, body) => {
+                if(error){
+                    if(error.code === 'ECONNREFUSED') { // Retry only if the ERROR is ERRCONNECT
+                        throw new Error(error.message);


If the error message is same, why not just throw the error itself?

Suggested change

throw new Error(error.message);

throw error;

coditva · 2021-10-06T12:36:01Z

lib/run/upload-run.js

+                    return bail(error); // For other errors , dont retry
+                }
+
+                // Handle exact status codes


We don't seem to be handling the exact status codes. 😅

Should use a switch-case in this case (pun intended.)

coditva · 2021-10-06T12:43:06Z

lib/run/upload-run.js

+ * @param {Object} uploadOptions
+ * @returns {function} Returns an async function which can be used by async-retry library to have retries
+ */
+_upload = (uploadOptions) => {


If this is going to return a function, it should be called something like _getUploadRetryable.
Or, don't return a function! Just bind the params where you're using it.

UmedJadhav added 6 commits September 25, 2021 20:17

Added upload with retry utility

db0ddfb

- This utility will be used to upload the newman runs to postman - We will be using exponential retry so as to not bombard the postman servers

Added unit tests for uploading newman runs

f814e5f

added a utility to extract resourceid from URL

02c0270

Integrated newman run upload in run flow

e956289

Fixed failing unit tests for upload run

c299ab9

Added new CLI params

63e3988

UmedJadhav added 3 commits September 27, 2021 17:16

Reverted the prettier changes

06a597a

minor spelling fixes

0de68dd

reverting minor prettier changes

79d4f87

UmedJadhav requested a review from codenirvana September 27, 2021 11:51

Minor changes around code structure

94578ec

coditva requested changes Sep 29, 2021

View reviewed changes

UmedJadhav added 5 commits October 4, 2021 11:24

Renamed the CLI parameters

76530b9

Refactored Upload Run

afa582a

Added Run Serialiser

Integrated refactored upload run in main flow

cadd6d9

Fixed the regex to fetch UID

53078af

Added tests for run serialiser

96bd6d4

coditva reviewed Oct 6, 2021

View reviewed changes

Removed unnecessary comments

506cb77

coditva requested changes Oct 6, 2021

View reviewed changes

Fixed upload run with a await

7a1b370

DannyDainton added the product/newman label Jul 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload newman runs to Postman #2849

Upload newman runs to Postman #2849

UmedJadhav commented Sep 27, 2021 •

edited

Loading

codecov bot commented Sep 27, 2021 •

edited

Loading

coditva Sep 29, 2021

UmedJadhav Oct 1, 2021

coditva Sep 29, 2021

UmedJadhav Oct 4, 2021

coditva Sep 29, 2021

giridharvc7 Oct 1, 2021

giridharvc7 Oct 1, 2021

coditva Sep 29, 2021

coditva Sep 29, 2021

coditva Oct 1, 2021

coditva Oct 1, 2021

UmedJadhav Oct 4, 2021

coditva Sep 29, 2021

coditva Sep 29, 2021

UmedJadhav Oct 4, 2021

coditva Sep 29, 2021

UmedJadhav Oct 4, 2021

coditva Sep 29, 2021

UmedJadhav Oct 1, 2021

coditva Sep 29, 2021

UmedJadhav Oct 4, 2021

coditva Oct 6, 2021 •

edited

Loading

coditva Oct 6, 2021

coditva Oct 6, 2021

coditva Oct 6, 2021

coditva Oct 6, 2021

coditva Oct 6, 2021

coditva Oct 6, 2021

	runError && !newmanStatus.resultUploaded && !_.get(options, 'suppressExitCode') && process.exit(1);
	(runError \|\| !newmanStatus.resultUploaded) && !_.get(options, 'suppressExitCode') && process.exit(1);


		// TODO: Add analytics call here

		if (err) { // TODO -> understand when this is raised

Upload newman runs to Postman #2849

Are you sure you want to change the base?

Upload newman runs to Postman #2849

Conversation

UmedJadhav commented Sep 27, 2021 • edited Loading

codecov bot commented Sep 27, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coditva Oct 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

UmedJadhav commented Sep 27, 2021 •

edited

Loading

codecov bot commented Sep 27, 2021 •

edited

Loading

coditva Oct 6, 2021 •

edited

Loading