U ]Cg+@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z dZ ej de je jddZdZd Zd d gZd d dZddZddZddZddZddZddZGdddejZedkredS)N)urljoin)ConfigzGhttps://github.com/ABI-Software/scicrunch-knowledge-testing/tree/doc_v1s3z us-east-1)Zaws_access_key_idZaws_secret_access_keyZ region_namezprd-sparc-discover50-use1z not-specifiedz&text/vnd.abi.plot+tab-separated-valuesztext/vnd.abi.plot+csvzimage/x.vnd.abi.thumbnail+jpegzimage/x.vnd.abi.thumbnail+png)z image/jpegz image/pngc CsVddi}dtji}tjd}||ddddd d d d d ddg d}tjt|d|||dS)Nacceptzapplication/jsonZapi_key/z item.curiez item.namez item.typesz objects.namezobjects.datacitezobjects.additional_mimetypezobjects.mimetypezobjects.datasetzpennsieve.versionzpennsieve.identifierz pennsieve.uri)fromsize_sourcez&_search?preference=abiknowledgetesting)jsonparamsheaders)rZSCICRUNCH_API_KEYZSCICRUNCH_API_HOSTrequestsZpostr)startrr r Zscicrunch_hostZscicrunch_requestrF/home/cmiss/Jenkins/workspace/Plot test/tests/slow_tests/plot_tests.py get_datasets%s&  rcCs|ddS)Nr)split) original_namerrrextract_bucket_name@src Cs|dd}d|krd|}|d|}z>tj||dd}|rZd|krZd|dd krZn |d d WSWn6tjjk r}z||d WYSd}~XYnXdS) Ndatasetpathfiles/rZ requester)ZBucketZKeyZ RequestPayerZResponseMetadataZHTTPStatusCodezInvalid response)ZS3PathReason)rZ head_objectbotocore exceptionsZ ClientError) dataset_idthumbnail_object s3_bucketZscicrunch_pathZ head_responseerrorrrrtest_plot_thumbnail_s3fileCs(  r!cCsxg}d}d}|dd}|dddt}d|krr5r6PlotzMissing version)r(S3_BUCKET_NAMErr:r8r-)rreportsourcerrBrr/Zobject_reportsrrrtest_datasets_informations6 &    rGcs$eZdZfddZddZZS)PlotDatasetFilesTestcstj||dS)N)super__init__)selfargskwds __class__rrrJszPlotDatasetFilesTest.__init__c Csd}d}d}d}d}d}ddggd}d}d} |rt||} |d| j| } |t| d d krld }||}| d d D]v} t| } d | kr| d r| d } td | dt| ddks| dddkr|d| d|d| q|t| d d }||kr.d }q.||d<| |d<td|dt|d|d<td|d|ddkr~td|dtj tj |ddt |d}tj ||ddW5QRXtd||dt|ddS)NrTzreports/plot_reports.jsonzreports/plot_path_mapping.json)TestedFailed FailedIdsDatasetsirhitsFrCzReports generated for r<r>r?r3rSrTrQzTested Datasets with PlotzNumber of datasets tested: rRzNumber of dataset with erros: zFailed Datasets: )exist_okw)indentz"Full report has been generated at )r assertEqualZ status_coder r9rGprintr-osmakedirsrdirnameopendump)rKrrZ keepGoingZ totalSizeZ reportOutputZpathMappingOutputZreportsZtestSizeZ totalPlotZscicrunch_responsedatarrEZoutfilerrrtest_files_informationsN.   z+PlotDatasetFilesTest.test_files_information)__name__ __module__ __qualname__rJrc __classcell__rrrNrrHs rH__main__)Zunittestr rZboto3r Z urllib.parseZurllibr]rerZ tests.configrZdoc_linkZclientZAWS_KEYZ AWS_SECRETrrDr)r7r*rrr!r1r:rGTestCaserHrdmainrrrrsB  6!j