U I>g#8 @s.ddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z iZ dZ dZ ejde je jddZd Zd Zd Zd Zd ZdZdZeeegZeeeeeeeeeeeed ZddZddZddZddZddZddZ ddZ!dd Z"d!d"Z#d#d$Z$Gd%d&d&ej%Z&e'd'kr*e(dS)(N)urljoin)ConfigzGhttps://github.com/ABI-Software/scicrunch-knowledge-testing/tree/doc_v1Fs3z us-east-1)Zaws_access_key_idZaws_secret_access_keyZ region_namez$pennsieve-prod-discover-publish-use1zabi-context-filezabi-plotzabi-scaffold-metadata-filezabi-scaffold-view-filez abi-thumbnailz not-specified) z.application/x.vnd.abi.context-information+jsonz(application/x.vnd.abi.scaffold.meta+jsonz(application/x.vnd.abi.scaffold.view+jsonzimage/x.vnd.abi.thumbnail+jpegzinode/vnd.abi.scaffold+filez inode/vnd.abi.scaffold+thumbnailz%inode/vnd.abi.scaffold.thumbnail+fileztext/vnd.abi.plot+thumbnailzinode/vnd.abi.plot+thumbnailz inode/vnd.abi.scaffold.view+filez&text/vnd.abi.plot+tab-separated-valuesztext/vnd.abi.plot+csvc Csnddi}dtji}tjd}||ddddd d d d d g d}trVdddtiii}||d<tjt|d|||dS)Nacceptzapplication/jsonZapi_key/z item.curiez item.namez item.typeszobjects.datacitezobjects.additional_mimetypezobjects.datasetzpennsieve.versionzpennsieve.identifierz pennsieve.uri)fromsize_sourcematchzpennsieve.identifier.aggregatequeryz&_search?preference=abiknowledgetesting)jsonparamsheaders)rZSCICRUNCH_API_KEYZSCICRUNCH_API_HOSTcheckDatasetOnlyrequestsZpostr)startrrr Zscicrunch_hostZscicrunch_requestr re/home/cmiss/Jenkins/workspace/Weekly SciCrunch Knowledge Test/tests/slow_tests/test_datasets_tests.py getDatasets4s4  rcCs|ddS)Nr)split) original_namerrrextract_bucket_nameYsrcCs4|dkr tS|tkrtS|}|tkr0t|StS)N) NOT_SPECIFIEDlowerTEST_MIME_TYPES) mime_typeZlower_mime_typerrr map_mime_type\src CszFtj||dd}|r0d|kr0d|ddkr0n||dtddWSWn>tjjk r}z|||td dWYSd}~XYnXdS) NZ requester)ZBucketZKeyZ RequestPayerZResponseMetadataZHTTPStatusCodezInvalid responsez#reason-invalid-response)MimetypePathReason ReasonDetailszM#reason-an-error-occurred-404-when-calling-the-headobject-operation-not-found)rZ head_objectdoc_linkbotocore exceptionsZ ClientError) localPathpathrbucketZ head_responseerrorrrrgetFileResponseks,  r+cCs"|dt}|tkr|d}|S)NZadditional_mimetypename)getr)objrrrrgetObjectMimeTypes  r/cstt|}|tkrdS|tkrd|krd|dkr|dd}d|krd|dkr|dd}d|krd|dkr|ddD]H}tj||tfddt|Dd}|rtt||rdSqd S) NTdatasetr(datacite isSourceOfrelativec3s&|]\}}|ddkr|VqdSr0r(Nr.0iitemZ actualPathrr sz$checkForThumbnail..F) rr/THUMBNAIL_IMAGESCAFFOLD_VIEW_FILEurllibparsernext enumerate)r.obj_listZlocal_mapped_typer'r2r(foundrr9rcheckForThumbnails   rCc sddd}dddggd}d}d|kr|D]}||dkr,|d|}d|kr,d|dkr,|ddD]} ||d ||<|d d 7<zztj|| tfd d t|Dd} | dkr||| d tdd|dd 7<n|dkrt|| |}Wqj||| dtdd|dd 7<YqjXqjq,|tkr|ddkrd|d<td|d<|dd 7<|dkrd|d<td|d<|dd 7<|S)Nr) isDerivedFromr2None) TotalErrorsThumbnailError ItemTestedrDr2Fr1r3r(rHc3s&|]\}}|ddkr|VqdSr4rr5r9rrr:sz$getDataciteReport..zCannot find the path#reason-cannot-find-the-path)Z RelativePathr"r#rFr2z*Encounter a problem while looking for pathz2#reason-encounter-a-problem-while-looking-for-pathzMissing isSourceOf entryrGz(#thumbnailerror-missing-issourceof-entryZThumbnailErrorDetailsz!Thumbnail not found in isSourceOfz1#thumbnailerror-thumbnail-not-found-in-issourceof) r=r>rr?r@appendr$rCMIMETYPE_WITH_THUMBNAILS) rAr.Zmapped_mimetypeZfilePathZ keysToCheckreportsZthumbnailFoundkeyZ keyObjectr(rBrr9rgetDataciteReportsT          rOc Csd}d}d|krzd|dkrz|dd}|d|} t|| ||}t||||}|ddkr|dkrp||d}||d<n|dd td d }|S) Nr0r(rrFr)r r!ZDataciteReportz Not foundzCannot find pathrJ)r r!r"r")r+rOr$) rAr.rmapped_mime_typeprefixr)ZdataciteReportZ fileResponser'r(rrrtestObjs&   rRcCsg}|d}d}d}g} |D]V} t| } t| } | tkrsf   %3/&5