a ,"?gƒçã @sò ddlZddlZddlmZddlmZmZmZddlm Z m Z ddl m Z m Z mZmZddlmZmZddlmZddlmZmZdd lmZddlZddlZddlZddlZddlZddlZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)ddl*m+Z+ddl,m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z;ddlm?Z?ddl@mAZAddlBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQddlRmSZSmTZTmUZUmVZVmWZWmXZXddlYmZZZddl[m\Z\ddl]m^Z^ddl_m`Z`maZambZbddlcmdZdmeZemfZfmgZgmhZhmiZimjZjdd lkmlZldd!lmmnZnmoZodd"lpmqZrdd#lpmsZtdd$lumdZvmwZxe y¡e/ezƒZ{e^j|e{j}d%<eAe{d&d'd(œd)Z~e4e{ƒe6e{ƒZeSƒZ€daej‚d*e^jƒe^j„d+d,Z…eZƒZ†ze`e^j‡ƒZˆWne‰ydZˆYn0zeae^j‡ƒZŠWne‰y:dZŠYn0zebe^j‡ƒZ‹Wne‰ybdZ‹Yn0Gd-d.„d.eŒƒZe{ Žd/¡d0d1„ƒZe{jd2d3„ƒZ‘e#ƒZ’e#ƒZ“e#ƒZ”e#ƒZ•e#ƒZ–e•j—sÌe ˜d4¡e• ™¡e^j|d5kre#ƒZšeƒZ›e› œ¡eš ™¡ešje›jœd6d7d7d8d9ddd:e^j|d;krTe^jždd?d@iZŸe{jdAdB„ƒZ iZ¡eƒZ¢e ƒZ£eƒZ¤e{jdCdD„ƒZ¥eƒZ¦e{jdEdF„ƒZ§e”je§dGdHdIe’je dGdHdIe“je¥dGdJdKe%e'ƒe)dHdLgƒZ¨e• dMdN„e¨¡dOdP„Z©e ªe©¡e{ «dQ¡dRdS„ƒZ¬e{j«dTdUgdVdWdX„ƒZ­dYdZ„Z®e{ «d[¡d\e^j¯fd]d^„ƒZ°e{ «d_¡d`da„ƒZ±e{ «db¡e^j¯fdcdd„ƒZ²e{ «de¡e^j¯fdfdg„ƒZ³dhdi„Z´e{ «dj¡dkdl„ƒZµdmdn„Z¶e{ «do¡e^j¯fdpdq„ƒZ·e{ «dr¡dsdt„ƒZ¸e{ «du¡e{ «dv¡dwdx„ƒƒZ¹e{ «dy¡dzd{„ƒZºe{ «d|¡d}d~„ƒZ»e{ «d¡e{ «d€¡dd‚„ƒƒZ¼e{ «dƒ¡e{ «d„¡d…d†„ƒƒZ½e{ «d‡¡e{ «dˆ¡d‰dŠ„ƒƒZ¾e{ «d‹¡dŒd„ƒZ¿e{ «dŽ¡dd„ƒZÀe{ «d‘¡d’d“„ƒZÁe{ «d”¡d•d–„ƒZÂe{ «d—¡e^j¯fd˜d™„ƒZÃe{ «dš¡d›dœ„ƒZÄddž„ZÅe{j«dŸd d¡dd¢œd£e{ «d¤¡d'd¥d¦„ƒƒZÆe{j«d§d¨d id£e{ «d©¡dªd«„ƒƒZÇe{ «d¬¡d­d®„ƒZÈd¯d°„ZÉd±d²„ZÊd³d´„ZËe{ «dµ¡d¶d·„ƒZÌe{ «d¸¡d¹dº„ƒZÍe{ «d»¡d¼d½„ƒZÎe{ «d¾¡d¿dÀ„ƒZÏe{ «dÁ¡dÂdăZÐe{j«dÄdÅgdVdÆdÇ„ƒZÑe{j«dÈdÅgdVdÉdÊ„ƒZÒe{j«dËdÅgdVe~jÓd'dÌdÍd΄ƒƒZÔe{j«dÏdÅgdVdÐdÑ„ƒZÕe{j«dÒdÅgdVdÓdÔ„ƒZÖe{j«dÕdÅgdVd(d×dØ„ƒZ×e{j«dÙdÅgdVdÚdÛ„ƒZØe{j«dÜdÅgdVdÝdÞ„ƒZÙe{j«dßdÅgdVdàdᄃZÚe{j«dâdÅgdVdãd䄃ZÛdådæ„ZÜdçdè„ZÝdédê„ZÞe{j«dëdUgdVdìd턃Zße{j«dîdUgdVdïdð„ƒZàe{j«dñdUgdVdòdó„ƒZáe{j«dôdUgdVdõdö„ƒZâe{j«d÷dUgdVdødù„ƒZãe{j«dúdUgdVdûdü„ƒZäe{j«dýdUgdVdþdÿ„ƒZåe{j«ddÅgdVdd„ƒZæe{ «d¡dd„ƒZçe{ «d¡dd„ƒZèe{ «d ¡d d „ƒZée{j«d dUgdVd d„ƒZêe{j«ddUgdVdd„ƒZqe{j«ddUgdVdd„ƒZse{j«ddUgdVdd„ƒZëe{ «d¡dd„ƒZìe{j«ddÅgdVdd„ƒZíe{j«ddÅgdVdd „ƒZîe{j«d!dUgdVd"d#„ƒZïe{j«d$dÅgdVd%d&„ƒZðdS()éN)Úget_download_count)Úinit_cf_cda_clientÚget_funded_projects_countÚget_featured_datasets)Úupdate_all_events_sort_orderÚupdate_event_sort_order)Úget_dataset_countÚinit_algolia_clientÚget_all_dataset_idsÚget_associated_datasets)Úinit_ga_reportingÚget_ga_1year_sessions)Ú MonthlyStats)Úset_featured_dataset_idÚ#get_featured_dataset_id_table_state)ÚOSparcServices)Úurlparse)ÚBackgroundScheduler)Ú OrTrigger)Ú DateTrigger)ÚIntervalTrigger)Ú ClientError)ÚdatetimeÚ timedelta)ÚFlaskÚabortÚjsonifyÚrequest)ÚCORS)Ú Marshmallow)Ú Pennsieve)ÚUnauthorizedException)ÚImage)Ú HTTPBasicAuth)ÚCache)Úcreate_doi_queryÚcreate_filter_requestÚcreate_facet_queryÚcreate_doi_aggregateÚcreate_title_queryÚcreate_identifier_queryÚ!create_pennsieve_identifier_queryÚcreate_field_queryÚcreate_request_body_for_curiesÚcreate_onto_term_queryÚcreate_multiple_doi_queryÚ create_multiple_discoverId_queryÚcreate_anatomy_queryÚget_body_scaffold_dataset_idÚcreate_multiple_mimetype_query)Ú EmailSenderÚfeedback_emailÚgeneral_interest_emailÚissue_reporting_emailÚ#creation_request_confirmation_emailÚservice_interest_email)ÚLock)Ú ElementTree)ÚConfig)ÚMapTableÚ ScaffoldTableÚFeaturedDatasetIdSelectorTable)Úprocess_resultsÚprocess_get_first_scaffold_infoÚreform_aggregation_resultsÚreform_curies_resultsÚreform_dataset_resultsÚreform_related_termsÚreform_anatomy_results)ÚContactRequestSchema)Úimg_to_base64_strÚget_path_from_mangled_list)Ústart_simulation)Úcheck_simulation)r@Úprocess_resultZENVZ SimpleCachei,)Z CACHE_TYPEZCACHE_DEFAULT_TIMEOUT)ÚconfigÚs3z us-east-1)Zaws_access_key_idZaws_secret_access_keyZ region_namec@s`eZdZdZe ¡edƒZdZe dd„ƒZ dd„Z e dd „ƒZ d d „Z e d d „ƒZdd„ZdS)Ú BiolucidaÚi?BFcCs |t_dS©N)rOÚ_token©Úvalue©rUú7/home/cmiss/Jenkins/workspace/SPARC-API-DEV/app/main.pyÚ set_tokenaszBiolucida.set_tokencCs|jSrQ)rR©ÚselfrUrUrVÚtokeneszBiolucida.tokencCs |t_dSrQ)rOÚ _expiry_daterSrUrUrVÚset_expiry_datehszBiolucida.set_expiry_datecCs|jSrQ)r[rXrUrUrVÚ expiry_datelszBiolucida.expiry_datecCs |t_dSrQ)rOÚ_pending_authenticationrSrUrUrVÚset_pending_authenticationosz$Biolucida.set_pending_authenticationcCs|jSrQ)r^rXrUrUrVÚpending_authenticationssz Biolucida.pending_authenticationN)Ú__name__Ú __module__Ú __qualname__rRrÚnowrr[r^Ú staticmethodrWrZr\r]r_r`rUrUrUrVrO\s   rOé”cCstt|ƒddfS)N)Úerrorrf)rÚstr)ÚerUrUrVÚresource_not_foundwsrjc CsÈzttjtjdtjdaWn¨tjjyV}z t   d¡t   |¡WYd}~nvd}~0t yŒ}z t   d¡t   |¡WYd}~n@d}~0t yÂ}z t   d¡t   |¡WYd}~n d}~00dS)NF)Z api_tokenZ api_secretZ env_overrideÚhostz#Unable to connect to Pennsieve hostz&Unable to authorise with Pennsieve Apiz Unknown Error) r r<ZPENNSIEVE_API_TOKENZPENNSIEVE_API_SECRETZPENNSIEVE_API_HOSTÚpsÚrequestsÚ exceptionsÚ HTTPErrorÚloggingrgÚPSUnauthorizedExceptionÚ Exception)ÚerrrUrUrVÚconnect_to_pennsieve|s ü    rtz6Starting scheduler for featured dataset id acquisitionÚ productionZcronÚ*Ú1Ú2)ÚyearÚmonthÚdayÚhourÚminuteÚsecondZ developmentZFALSEz8Starting scheduler for updating contentful event entrieséz US/Eastern)r|Útimezonec Cs¤t d¡zLtjtj›dd}|jrTd|j dd¡vrT| ¡}t |dƒ}|t d<Wn0t y†}zt  d |¡WYd}~n d}~00t js t d ¡t  ¡dS) NzGetting oSPARC viewersz/viewers©Úurlúapplication/jsonú content-typerPÚdataÚ file_viewersz!Could not retreive oSPARC viewersz1Starting scheduler for oSPARC viewers acquisition)rpÚinformÚgetr<ÚOSPARC_API_HOSTÚokÚheadersÚjsonÚbuild_filetypes_tableÚ osparc_datarrrgÚviewers_schedulerÚrunningÚstart)ÚreqZviewersÚtablerirUrUrVÚget_osparc_file_viewers³s   " r”cCspt d¡trttƒ}|td<tr2ttƒ}|td<trFttƒ}|td<t ƒ}|td<t j slt d¡t   ¡dS)NzGathering metrics dataZ1year_sessions_countZ dataset_countZfunded_projects_countZ1year_download_countz*Starting scheduler for metrics acquisition) rpr‡Úgoogle_analyticsr Ú usage_metricsÚalgoliarÚ contentfulrrÚmetrics_schedulerrr‘)Z ga_responseZalgolia_responseZ cf_responseZ ps_responserUrUrVÚ get_metricsÊs  ršc Csnt d¡z.tjtj›dd}| ¡}t |d¡Wn0t yh}zt  d|¡WYd}~n d}~00dS)NzFetching oSPARC servicesz /servicesrr…z%Request to get oSPARC services failed) rpr‡rmrˆr<r‰rŒÚosparc_servicesZ set_servicesrrrg)r’Z services_resprirUrUrVÚ get_servicesãs rœÚintervalé)ÚfuncÚtriggerÚdaysé)rŸr Úhours)r£cCsttƒSrQ)rÚfeaturedDatasetIdSelectorTablerUrUrUrVÚùór¥cCs|t d¡tjrt ¡t d¡tjr0t ¡t d¡tjrHt ¡t d¡tjr`t ¡t d¡tjrxt ¡dS)Nz1Stopping scheduler for oSPARC viewers acquisitionz*Stopping scheduler for metrics acquisitionz2Stopping scheduler for updating contentful entriesz3Stopping scheduler for updating featured dataset idz&Stopping scheduler for oSPARC services) rpr‡rrÚshutdownr™Ú)update_contentful_event_entries_schedulerÚfeatured_dataset_id_schedulerÚservices_schedulerrUrUrUrVÚshutdown_schedulersûs     r«z/healthcCst ddi¡S)NÚstatusZhealthy)rŒÚdumpsrUrUrUrVÚhealthsr®z/contactÚPOST)Úmethodsc Csht tj¡}tƒ |¡}|d}|d}|d}t |||¡t t j |dt   d|i¡¡t  ddi¡S)NÚnameÚemailÚmessagezFeedback submissionr¬Úsent)rŒÚloadsrr…rGÚloadÚ email_senderZ send_emailÚsendgrid_emailr<Ú SES_SENDERr5Ú substituter­)r…Zcontact_requestr±r²r³rUrUrVÚcontacts  r»cCstjd||d|dœ|d}|S)NÚ get_objectÚ requester)ÚBucketÚKeyÚ RequestPayerZResponseContentType)ZParamsZ ExpiresIn)rNZgenerate_presigned_url)Ú s3BucketNameÚkeyÚ content_typeÚ expirationÚresponserUrUrVÚcreate_s3_presigned_url#s  ýrÆz /downloadicCs6tj d¡}tj d|¡}tj dd¡}t||||ƒS)NrÂrÁZ contentTypezapplication/octet-stream)rÚargsrˆrÆ)rÄÚ bucket_namerÂrÁrÃrUrUrVÚcreate_presigned_url.s rÉz/thumbnail/neurolucidac Cstj}d|vsd|vsd|vr*tdddStj›d}zPtj||dd }| ¡|jd krz|j  d d ¡d krzt   |j ¡WStddƒWn|tj jy¨tdddYStj jyÆtddƒYStj jy}z tddt|ƒ›ƒWYd}~Sd}~00dS)NÚversionZ datasetIdÚpathéúQuery arguments are not valid.©Ú descriptionz /thumbnailé)ÚparamsÚtimeoutéÈz Content-TypeÚunknownz image/pngzFailed to retrieve thumbnail.z0Unable to make a connection to NEUROLUCIDA_HOST.iøz&Request to NEUROLUCIDA_HOST timed out.éöz)Error while requesting NEUROLUCIDA_HOST: )rrÇrr<ZNEUROLUCIDA_HOSTrmrˆZraise_for_statusÚ status_coder‹Úbase64Ú b64encodeÚcontentrnÚConnectionErrorZTimeoutZRequestExceptionrh)Ú query_argsr‚rÅrirUrUrVÚthumbnail_from_neurolucida_file7s"   rÜz/thumbnail/segmentationc Cstj}d|vrtdddS| d|¡}|d}d}d}d}d}d }|} |rP|sTz"tj||d |›d | ›d d } Wnvtyè} z^| jdddkr´tdd|›ddWYd} ~ Stdd|›ddWYd} ~ SWYd} ~ n d} ~ 00| d ¡  d¡}d|v}d|v}|r"|s"| |7} n||7}| |7} t |ƒ|krFtdd|›ddSqF|durrtdd|›ddS||  d¡d…} | d|   d¡…d} t   | ¡}|j}d}|D]}||jdd…7}q²t |¡}t dt|dƒt|dƒf|¡}t|ƒ}|S)z° Extract a thumbnail from a mbf xml file. First phase is to find the thumbnail element in the xml document. Second phase is to convert the xml to a base64 png. rËrÌrÍrÎrÁNFrièzbytes=ú-r½)r¾r¿ZRangerÀÚErrorÚCodeÚ NoSuchKeyrfúCould not find file: 'ú'úUnknown error for file: 'ÚBodyzUTF-8z z#Could not find thumbnail in file: 'rPrZRGBZrowsÚcols)rrÇrrˆrNr¼rrÅÚreadÚdecodeÚlenÚfindr;Ú fromstringÚattribÚtextÚbytesÚfromhexr"Ú frombytesÚintrH)rÈrÛrÁrËÚresourceZstart_tag_foundZ end_tag_foundZ start_byteÚoffsetZend_byterÅÚexZstart_thumbnail_elementZ thumbnail_xmlÚxmlZ size_infoZim_dataÚchildZ byte_im_dataZimZ base64_formrUrUrVÚextract_thumbnail_from_xml_fileNsZ   ü  6      "röz/exists/cCsftj}| d|¡}ztj||dd}Wnty@ddiYS0| dd¡}|dkr^ddiSddiS) NrÁr½©r¾r¿rÀÚexistsÚfalseÚ ContentLengthrÚtrue)rrÇrˆrNÚ head_objectr)rËrÈrÛrÁÚ head_responseÚcontent_lengthrUrUrVÚ url_existss ý   rÿcCs0| dd¡}|d| d¡…}t |¡}| ¡S)Nzhttps://api.pennsieve.io/z"https://api.pennsieve.io/discover/ú/)ÚreplaceÚrfindrmrˆrŒ)ÚuriÚrrUrUrVÚfetch_discover_file_information¤s  rz/s3-resource/discover_pathc Cs†tj d¡}z6t|ƒ}d|vr@|ddkr@|dd}|dWSWn0tyr}zt d|¡WYd}~n d}~00tdd |›d S) NrZ totalCountržÚfilesrrËzFailed to retrieve uri {uri}rfzFailed to retrieve uri rÎ)rrÇrˆrrrrprgr)rZ json_responseÚ file_inforórUrUrVÚget_discover_path­s  "rc CsäzDtj||dd}| dtj¡}|rB|tjksBtdd|›dWSWn–tjjyÚ}zz|j ddd krzdWYd}~S|j ddd krœdWYd}~St|j dd|j ddƒWYd}~SWYd}~nd}~00dSdS)Nr½r÷rúizFile too big to download: rÎrÞrßZ404rfú.Provided path was not found on the s3 resourceZ403é“úEThere is a permission issue when accessing the file at specified pathÚMessage)rÓÚOK)rfr )r r ) rNrürˆr<ZDIRECT_DOWNLOAD_LIMITrÚbotocorernrrÅ)rËrÈrýrþrsrUrUrVÚs3_header_checkºs ý@rz/s3-resource/c Csàtj}| d|¡}|}t||ƒ}|ddks8|ddkr¢t|ƒ}||krTtdddt||ƒ}|ddkrp|}n2|ddkrŠtdddn|ddkr¢tdddtj||d d }tj d ¡}|d  ¡} |durÜt   | ¡S| S) NrÁrrfr r rÎrÓr r½r÷Z encodeBase64rä) rrÇrˆrrIrrNr¼rær×rØ) rËrÈrÛrÁZs3_pathrÅZs3_path_modifiedZ response2Ú encode_base64rñrUrUrVÚdirect_download_urlÓs2        ý   rz /scicrunch-dataset//c Csˆ| dd¡d|}t|ƒ}z&tjtj›dtj›|d}| ¡WStjj y‚}z$t   |¡t  d|i¡WYd}~Sd}~00dS)NzDOI:rPrú/_search?api_key=©rŒrg) rr%rmÚpostr<ÚSCI_CRUNCH_HOSTÚKNOWLEDGEBASE_KEYrŒrnrorprgr­)Zdoi1Zdoi2Údoir…rÅrsrUrUrVÚsci_doiûsþ  rz /pubmed/z/pubmed//c Cs`zt d|›d¡}|jWStjjyZ}z$t |¡t d|i¡WYd}~Sd}~00dS)Nz https://pubmed.ncbi.nlm.nih.gov/rrg) rmrˆrìrnrorprgrŒr­)Úid_rÅrsrUrUrVÚpubmed s  rz/scicrunch-query-string/c Cs®tj d¡}tj d¡}tj d¡}tj d¡}t||||ƒ}z*tjtj›dtj ›|d}t |  ¡ƒWStj j y¨}z$t |¡t  d|i¡WYd}~Sd}~00dS)NÚfieldÚcurieÚsizeÚfromrrrg)rrÇÚgetlistrˆr,rmrr<rrr@rŒrnrorprgr­)ÚfieldsrrZfrom_r…rÅrsrUrUrVÚ sci_organs    þ r!z/dataset_info/using_doicCs<tj d¡}tj d¡}t|ƒ}|dur4tt|ƒƒSt|ƒS)NrZ raw_response)rrÇrˆr%rDÚdataset_search)rÚrawÚqueryrUrUrVÚget_dataset_info_doi*s    r%z!/dataset_info/using_multiple_doisz"/dataset_info/using_multiple_dois/cCs tj d¡}t|ƒ}tt|ƒƒS)NÚdois)rrÇrr/r@r")r&r$rUrUrVÚget_dataset_info_dois5s r'z./multiple_dataset_info/using_multiple_mimetypez//multiple_dataset_info/using_multiple_mimetype/cCs tj d¡}t|ƒ}tt|ƒƒS)NÚq)rrÇrr3r@r")r(r$rUrUrVÚget_file_info_from_mimetype=s r)z(/dataset_info/using_multiple_discoverIdsz)/dataset_info/using_multiple_discoverIds/cCs tj d¡}t|ƒ}tt|ƒƒS)NÚ discoverIds)rrÇrr0r@r")r*r$rUrUrVÚget_dataset_info_discoverIdsGs r+z/dataset_info/using_titlecCs tj d¡}t|ƒ}tt|ƒƒS)NÚtitle)rrÇrˆr)rDr")r,r$rUrUrVÚget_dataset_info_titlePs r-z%/dataset_info/using_object_identifiercCs tj d¡}t|ƒ}tt|ƒƒS©NÚ identifier)rrÇrˆr*rDr"©r/r$rUrUrVÚ"get_dataset_info_object_identifierXs r1z/dataset_info/anatomycCs6tj dd¡}|dkr"tdddSt|ƒ}tt|ƒƒS)Nr/éÿÿÿÿrfz Identifier for API call not set.rÎ)rrÇrˆrr1rFr"r0rUrUrVÚget_dataset_info_anatomy`s  r3z(/dataset_info/using_pennsieve_identifiercCs tj d¡}t|ƒ}tt|ƒƒSr.)rrÇrˆr+rDr"r0rUrUrVÚ%get_dataset_info_pennsieve_identifierks r4z/segmentation_info/c CsZtj}d|vrtdddS| d|¡}| d¡}z6t||ƒ}|ddkrTtdddtj||d d }WnvtyÜ}z^|jd d d kr¨tdd|›ddWYd}~Stdd|›ddWYd}~SWYd}~n d}~00|d  ¡}t   |¡}|  d¡}i} |dur|j | d<ndddddœ| d<|  d¡} | durJ| j | d<n ddi| d<| S)NÚ dataset_pathrÌz,Query arguments must include 'dataset_path'.rÎrÁrrfr r½r÷rÞrßràrárârãräz./{*}sparcdata/{*}subjectÚsubjectrP)ZageZsexÚspeciesZ subjectidz./{*}sparcdata/{*}atlasZatlasZorgan)rrÇrrˆrrNr¼rrÅrær;rêrérë) rÈrÛrÁr5rÅrórñrôZsubject_elementr‡Z atlas_elementrUrUrVÚget_segmentation_info_from_filess<      ý  6         r8z/current_doi_listcCsBtƒ}tt|ƒƒ}g}|ddD]}| |dd¡q"d|iS)NrÚbucketsrÂrÚresults)r(rBr"Úappend)r$r:Z doi_resultsÚresultrUrUrVÚ get_all_doi s  r=c Cstz0|}dtji}tjtj›d||d}| ¡WStjjyn}z"t  |¡t d|iƒWYd}~Sd}~00dS)NÚapi_keyú/_search)rŒrÑrg) r<rrmrrrŒrnrorprgr)r$ÚpayloadrÑrÅrsrUrUrVr"«sÿÿ  r"z/search/rPé )r$Úlimitr‘)Údefaultsz/search/c CsÒzŒtj d¡durtj d¡}tj d¡dur:tj d¡}tj d¡durVtj d¡}t tj›d|›d|›d|›dtj› ¡}t| ¡ƒWStj j yÌ}z$t   |¡t  d|i¡WYd}~Sd}~00dS) NrBr$r‘z /_search?q=z&size=z&from=z &api_key=rg)rrÇrˆrmr<rrr@rŒrnrorprgr­)r$rBr‘rÅrsrUrUrVÚ kb_search½s   * rDz/filter-search/r$z/filter-search//c CsÞtj d¡}tj d¡}tj d¡}tj d¡}t|||||ƒ}z,tjtj›dtj ›|d}t |  ¡ƒ}Wnltj j y´}z,t |¡tt|ƒddœƒd fWYd}~Sd}~0t jyØtd d d œƒd fYS0|S) NÚtermZfacetrr‘rrzc Csòt|ƒ\}}g}||D]¬}||d|›dd<z2tjtj›dtj›|d}| ¡}| |¡WqtjyŽt dddœƒd fYSt yÂ}zt   d |›|¡WYd}~qd}~00qg}|D]} || d|›d 7}qÎt |ƒS) NZ aggregationsrJrrrrGrHrIrÕz$Could not search SciCrunch for path r9) r'rmrr<rrrŒr;rHrrrrprg) Útype_Ztype_mapr…r:rËrÅZ json_resultrórJr<rUrUrVÚ get_facetsìs.  þÿÿ *rMcCs&d|vr"t | d¡¡}|j|d<dS)NZreadmeZmarkdown)rmrˆrì)ÚrespZmark_reqrUrUrVÚinject_markdown srOc CsB| d¡}| d¡}|dus$|dur(dSt|ƒ}|j}ztj|d |¡dd}Wn|tyÌtjdk}|rvt   d¡ztj|d |¡dd}Wn6tyÆ}z|r®t   |¡WYd}~YdSd}~00Yn0|d   ¡}zt  |¡} Wn2ty}zt   |¡WYd}~dSd}~00|  d ¡|  d ¡|  d ¡d œ|d<dS)NÚidrz{}/files/template.jsonr½r÷ZTRUEzXRequired file template.json was not found under /files folder, trying under /packages...z{}/packages/template.jsonräÚuuidr±rÏ)rQr±rÏZstudy)rˆrÚnetlocrNr¼Úformatrr<ÚSPARC_API_DEBUGGINGrpÚwarningrgrærŒrµÚ ValueError) rNrrZ parsed_uriÚbucketrÅÚ debuggingriÚtemplateZ template_jsonrUrUrVÚinject_template_datasL  ý   ÿý     ýrZcCsFi}|D]8}|d ¡}|d=| |d¡s2g||<|| |¡q|S)NZ file_typeF)Úlowerrˆr;)Zosparc_viewersr“ZviewerZfiletyperUrUrVr@s  rz/sim/dataset/c Cs’tjdkrŽz@t d tj|¡¡}|jrH| ¡}t |ƒt |ƒt |ƒWSWn6t y€}zt  d|›|¡WYd}~n d}~00tdddSdS)NÚGETz{}/datasets/{}úCould not fetch SIM dataset rfúResource not foundrΩrÚmethodrmrˆrSr<ÚDISCOVER_API_HOSTrŠrŒrOrZrrrrprgr)rr’Ú json_datarórUrUrVÚ sim_datasetJs (rcz&/sim/dataset//versions/c Csštjdkr–zBt d tj||¡¡}|jrJ| ¡}t |ƒt |ƒt |ƒWSWn<t yˆ}z$t  d|›d|›|¡WYd}~n d}~00tdddSdS)Nr\z{}/datasets/{}/versions/{}r]z version rfr^rÎr_)rZversion_r’rbrórUrUrVÚsim_dataset_versionsXs .rdz/get_osparc_datacCsttƒSrQ)rrŽrUrUrUrVÚget_osparc_datahsrez /sim/servicecCsTtjdkrPtj d¡}tjjddtd}tjjddtd}t |||¡}t|ƒSdS)Nr\ÚsearchrBrÐ)ÚdefaultÚtypeÚskipr)rr`rÇrˆrðr›Zsearch_servicesr)rfrBrir:rUrUrVÚ osparc_searchms   rjz /sim/filecCs tjdkrtj}td|iƒSdS)Nr\r†)rr`r›Zfile_extensionsr)Ú extensionsrUrUrVÚosparc_extensionsws  ÿrlz/project/r\cCs4t|ƒ}t|dƒdkr$t|dƒStddddS)NÚhitsrrfr^rÎ)r rèrr)Z project_idÚdatasetsrUrUrVÚdatasets_by_project_id€s roz"/get_featured_datasets_identifierscCs dtƒiS)NÚ identifiers)rrUrUrUrVÚ!get_featured_datasets_identifiersŠsrqz/get_featured_dataset)rÒc Cs¢ttƒd}|dkrd}zBt d tj|¡¡ ¡}|dgkrVt d tjd¡¡ ¡}|WSty}zt   d|›|¡WYd}~n d}~00t ddd dS) NÚfeatured_dataset_idr2é z{}/datasets?ids={}rnzCould not get featured dataset rfz,An error occured while fetching the resourcerÎ) rr¤rmrˆrSr<rarŒrrrprgr)rrrÅrórUrUrVÚget_featured_datasetŽs  (rtz/get_owner_email/csPtjj}tjj |¡}‡fdd„|Dƒ}|s:tdddntd|djiƒSdS)Ncsg|]}|jˆkr|‘qSrU)Zint_id)Ú.0Úx©Úowner_idrUrVÚ ¥r¦z#get_owner_email..rfzOwner not foundrÎr²r)rlZ_apiZ _organizationZ organizationsZ get_membersrrr²)rxZorgÚmembersÚresrUrwrVÚget_owner_email s r|z!/get_body_scaffold_info/cCs:t|ƒ}|r(t|ƒ}tt|ƒƒ}|r(|Stdd|›dS)NrfzWhole body info not found for rÎ)r2r+rAr"r)r7rPr$r<rUrUrVÚget_body_scaffold_info¯s r}z/thumbnail/Fc Csütƒ}z´t| ¡stƒWdƒn1s00Ytjd |¡}d| ¡i}tjd||d}t   |j ¡}|dkr¶|s¶t|  d¡Wdƒn1s¢0Yt |dƒ}|WStyð}zt d|›|¡WYd}~n d}~00td d ƒS) Nz/thumbnail/{0}rZr\©r‹s¼eyJzdGF0dXMiOiJBZG1pbiB1c2VyIGF1dGhlbnRpY2F0aW9uIHJlcXVpcmVkIHRvIHZpZXcvZWRpdCB1c2VyIGluZm8uIFlvdSBtYXkgbmVlZCB0byBsb2cgb3V0IGFuZCBsb2cgYmFjayBpbiB0byByZXZlcmlmeSB5b3VyIGNyZWRlbnRpYWxzLiJ9rPTz Could not get the thumbnail for rfz-An error occured while fetching the thumbnail)rOÚbiolucida_lockrZÚauthenticate_biolucidar<ÚBIOLUCIDA_ENDPOINTrSrmrr×rØrÙrWÚthumbnail_by_image_idrrrprgr)Úimage_idZrecursive_callÚblr‚r‹rÅZencoded_contentrórUrUrVr‚ºs($ÿ ÿ( (r‚z/image/c Csltjd |¡}zt d|¡}t| ¡ƒWSty`}zt  d|›|¡WYd}~n d}~00t ddƒS)Nz/image/info/{0}r\zCould not get image info for rfz/An error occured while getting the image's info) r<rrSrmrÚprocess_biolucida_resultrŒrrrprgr)rƒr‚rÅrórUrUrVÚimage_info_by_image_idÚs (r†z/image_search/c Csjtjd |¡}zt d|¡}| ¡WSty\}zt d|›|¡WYd}~n d}~00ddidfS)Nz%/imagemap/search_dataset/discover/{0}r\z$Could not search images for dataset rgz3An error occured while searching images for datasetrf) r<rrSrmrrŒrrrprg)Z dataset_idr‚rÅrórUrUrVÚimage_search_by_dataset_idås  (r‡z/image_xmp_info/cCsvtjd |¡}zt d|¡}Wn"tjjyBtdddYS0| ¡}|ddkrdt |dƒStdd |›dS) Nz/image/xmpmetadata/{0}r\rÌú)Unable to make a connection to Biolucida.rÎr¬Úsuccessr…zXMP info not found for ) r<rrSrmrrnrÚrrŒÚprocess_biolucida_results©rƒr‚r<rÅrUrUrVÚimage_xmp_infoðs  rŒz/image_blv_link/cCsztjd |¡}zt d|¡}Wn"tjjyBtdddYS0| ¡}|ddkrht d|diƒStdd |›dS) Nz/image/blv_link/{0}r\rÌrˆrÎr¬r‰ÚlinkzBLV link not found for ) r<rrSrmrrnrÚrrŒrr‹rUrUrVÚimage_blv_linkÿs rŽcCsdtƒ}tjd}tjtjddœ}g}i}tjd||||d}|jtjj kr`|  ¡}|  |d¡dS)Nz /authenticaterP)ÚusernameÚpasswordrZr¯)r‹r…rrZ) rOr<rZBIOLUCIDA_USERNAMEZBIOLUCIDA_PASSWORDrmrrÖZcodesrŠrŒrW)r„r‚r@rr‹rÅrÙrUrUrVr€s þr€cCshd}tjdrd}|rXt ¡}|rJd|vrJ|d}| ||¡}td|iƒStdddn td d ddS) NTÚTESTINGFÚstaterQrÌzState not specifiedrÎrfúDatabase not available)ÚapprMrÚget_jsonZ pushStaterr)r“Zcommitrbr’rQrUrUrVÚget_share_links    r–cCs^|rNt ¡}|r@d|vr@|d}| |¡}|r@td| |¡iƒStdddn tddddS)NrQr’rÌz#Key missing or did not find a matchrÎrfr“)rr•Z pullStaterr)r“rbrQr’rUrUrVÚget_saved_state/s  r—z/map/getshareidcCsttƒSrQ)r–ÚmaptablerUrUrUrVÚget_map_share_link=sr™z /map/getstatecCsttƒSrQ)r—r˜rUrUrUrVÚ get_map_stateCsršz/scaffold/getshareidcCsttƒSrQ)r–Ú scaffoldtablerUrUrUrVÚget_scaffold_share_linkIsrœz/scaffold/getstatecCsttƒSrQ)r—r›rUrUrUrVÚget_scaffold_stateOsrú/tasksc# CsJtj}d|vrˆzFtjtjtj|ddœd}| ¡}d|vsD|dsRddidfWSWqžty„}zt   d|¡WYd}~qžd}~00nt j d sžddidfS|r:d |vr:d |vr:|d }|d }|d }d d tj i}dtjd}tjtjtjtjtjtjg} tjtjtjtjtjtjg} tj} d} d} g}|rNd|vrN|d} | dkr’dtjd}tjtjtjg} tjtjtjg} tj} tj} | dkrØdtjd}tjtjtjg} tjtjtjg} tj} tj} nº| dkrdtjd}tjtjtjg} tjtjtjg} tj} tj} nt| dkrddtjd}tjtjtjg} tjtjtjg} tj} tj } n.| dkr’|  !tj"tj#g¡|  !tj"tj#g¡| dkrþd| }tj$||d}d| ¡vrþ| ¡dgkrþ| ¡ddd |}| ¡ddd}||| | | dddidœ}tj|||d}tj%}d| ¡vr®| ¡dgkr®| ¡ddd }|rðd!|vrð|d!}| &¡}|j'}|j(}d tj ||d"d#œ}d|d$}ztj|||d%Wn.tyî}zt)|ƒWYd}~n d}~00|D]¸}d|}tj$||d}d| ¡vrô| ¡dgkrô| ¡ddd | ¡ddd | ¡ddd&| ¡ddd'| ¡ddd(d|gddid)œ}tj|||dqô|j*d*kr0d+|vr|d+rd,|vr|d,d-krd.} t+ ,d/|i¡}!| d0krd1} t- ,d/|i¡}!nÐ| d2kr4d3} t. ,d/|i¡}!n²| d4krRd5} t/ ,d/|i¡}!n”| dkrpd6} t0 ,d/|i¡}!nv| dkrŽd7} t0 ,d/|i¡}!nX| dkr¬d8} t0 ,d/|i¡}!n:| dkrÊd9} t0 ,d/|i¡}!n| dkræd:} t0 ,d/|i¡}!|d+}"t1|"ƒdkrt2 3tj4|d+| |!¡t5||| ¡ddd d;S| ¡Sn t6ddS)?NZ captcha_token)ZsecretrÅ)r‚rŒr‰rgzFailed Captcha Validationi™z0Could not validate captcha, bypassing validationr‘r,rÏÚ AuthorizationzBearer z%https://www.wrike.com/api/v4/folders/ržrPrhÚnewsÚeventZtoolsAndResourcesZcommunitySpotlightZresearchz#https://www.wrike.com/api/v4/tasks/©r‚r‹r…rZ subTaskIdsFZBacklog)r,rÏÚ customStatusÚ followersÚ responsiblesÚfollowÚdates)r‚rŒr‹rPÚ attachmentZXMLHttpRequest)rŸz X-File-Namer„zX-Requested-Withz /attachments)r‚r…r‹ZcustomStatusIdZ followerIdsZresponsibleIds)r,rÏr£r¤r¥r¦Z superTasksr§rÓÚ userEmailZsendCopyrûz%SPARC Reported Error/Issue Submissionr³ZfeedbackzSPARC Feedback SubmissionZinterestz!SPARC Service Interest SubmissionÚgeneralz$SPARC Question or Inquiry SubmissionzSPARC Research SubmissionzSPARC News SubmissionzSPARC Event SubmissionzSPARC Tool/Resource SubmissionzSPARC Story Submission)r,rÏÚtask_idrÌzMissing title or descriptionrÎ)7rÚformrmrr<Z TURNSTILE_URLZNUXT_TURNSTILE_SECRET_KEYrŒrrrprgr”rMÚ WRIKE_TOKENZDRC_FEEDBACK_FOLDER_IDZCCB_HEAD_WRIKE_IDZDAT_CORE_TECH_LEAD_WRIKE_IDZMAP_CORE_TECH_LEAD_WRIKE_IDZK_CORE_TECH_LEAD_WRIKE_IDZSIM_CORE_TECH_LEAD_WRIKE_IDZMODERATOR_WRIKE_IDZDRC_WRIKE_CUSTOM_STATUS_IDZNEWS_AND_EVENTS_FOLDER_IDZCOMMS_LEAD_1_WRIKE_IDZCOMMS_LEAD_2_WRIKE_IDZCOMMS_LEAD_3_WRIKE_IDZCOMMS_WRIKE_CUSTOM_STATUS_IDZNEWS_TEMPLATE_TASK_IDZEVENT_TEMPLATE_TASK_IDZTOOLS_AND_RESOURCES_FOLDER_IDZ$TOOLS_AND_RESOURCES_TEMPLATE_TASK_IDZCOMMUNITY_SPOTLIGHT_FOLDER_IDZ$COMMUNITY_SPOTLIGHT_TEMPLATE_TASK_IDÚextendZ SUE_WRIKE_IDZ JYL_WRIKE_IDrˆrræÚfilenamerÃÚprintrÖr7rºr5r9r6r8rèr·r¸r¹rr)#r¬Z captchaReqZ captchaResprór,rÏZnewTaskDescriptionÚhedr‚r¤r¥r£ZtaskTypeZtemplateTaskIdZtemplateSubTaskIdsZ templateUrlZ templateRespr…rNrZ new_task_idr¨Z file_dataÚ file_namerÃr‹Zattachment_urlriZ subTaskIdZsubTaskTemplateUrlZsubTaskTemplateRespZ subTaskDatar6Úbodyr©rUrUrVÚcreate_wrike_taskTs@þþ$        þ ù ý ü ý þ ø ý  ,        ý r´z/mailchimp_subscribec Csæt ¡}|rÖd|vrÖd|vrÖd|vrÖ|d}|d}|d}tdtjƒ}d|}|d||dœdœ}z6tj|||d }|jd krˆ| ¡WSd | ¡WSWn0t yÈ}zt   d |¡WYd}~n d}~00d didfSt ddddS)NÚ email_addressÚ first_nameÚ last_nameÚAnyUserú;https://us2.api.mailchimp.com/3.0/lists/c81a347bd8/members/Z subscribed)ZFNAMEZLNAME)rµr¬Z merge_fields©r‚rŒÚauthrÓz(Failed to subscribe user with response: z!Could not subscribe to newsletterrgzAn error occured while trying to unsubscribe to the newsletterr¼rÌúMissing email_addressrÎr½)rbrµr»r‚r…rNrórUrUrVÚunsubscribe_to_mailchimp0s(  ÿý  " rÂz&/mailchimp_member_info/c Cs |rtdtjƒ}d|}z4tj||d}|jdkr<| ¡WSd| ¡WSWn6ty‚}zt  d|›|¡WYd}~n d}~00ddid fSt d d d dS) Nr¸r¹)r‚r»rÓz)Failed to get member info with response: zFailed to get member info for rgz(Could not get member info from MailChimpr¼rÌrÁrÎ) r#r<r¾rmrˆrÖrŒrrrprgr)rµr»r‚rNrórUrUrVÚget_mailchimp_member_infoLs þ  ( rÃz/get-organ-curies/c CsŒtj d¡}t|ƒ}i}z,tjtj›dtj›|d}t |  ¡ƒ}Wn>t y‚}z&t   d|¡dddœdfWYd}~Sd}~00t|ƒS) Nr7rrzFailed getting Uberon IDsrGÚ BaseExceptionrIrÕ)rrÇrr-rmrr<rrrCrŒrÄrprgr)r7Z requestBodyr<rÅrórUrUrVÚget_available_uberonidsds" þ þýrÅz/get-related-terms/c Cs°tjjdddtjjdddtjjdddtjdœ}i}z*tjtj›d |›|d }t| ¡ƒ}WnDt y¦}z,t   d |›|¡d d dœdfWYd}~Sd}~00t |ƒS)NÚ directionZOUTGOING)rgÚrelationshipTypez BFO:0000050Úentailrû)rÆrÇrÈr>z/graph/neighbors/)rÑz*Failed getting related terms with payload rGrÄrIrÕ) rrÇrˆr<rrmZSCI_CRUNCH_SCIGRAPH_HOSTrErŒrÄrprgr)r$r@r<rÅrórUrUrVÚget_related_terms|s(üþþýrÉz /simulation_ui_file/cCs¤ttt|ƒƒƒ}t |j¡}zd|dd}|d}|dddd}t dd|›d |›¡}t d d|¡}tt t ||ƒ¡ƒWSt yžt d d d Yn0dS)Nr:rZs3urizabi-simulation-fileZdatasetrËz s3://[^/]*/rPzfiles/z s3://|/.*rfz$no simulation UI file could be foundrÎ) r@r"r+rŒrµr…ÚreÚsubrrrrr)r/r:Z results_jsonÚitemrrËrÂZs3_bucket_namerUrUrVÚsimulation_ui_file–s   rÍz /pmr_filecCs€t ¡}|rpd|vrpz@t tj›d|d›¡}|jdkrHt |j ¡WS|  ¡WSWq|t dddYq|0n t ddddS)NrËrrÓrÌz invalid pathrÎz missing path) rr•rmrr<ZPMR_HOSTrÖr×rØrÙrŒr)r…rNrUrUrVÚpmr_file¨s  rÎz/start_simulationcCsJt ¡}|r:d|vr:d|dvr:d|dvr:t t|ƒ¡StddddS)NÚsolverr±rÊrÌz)Missing solver name and/or solver versionrÎ)rr•rŒr­Údo_start_simulationr©r…rUrUrVrJ¸s$rJz/check_simulationcCsRt ¡}|rBd|vrBd|vrBd|dvrBd|dvrBt t|ƒ¡StddddS)NZjob_idrÏr±rÊrÌz1Missing solver name, solver version and/or job idrÎ)rr•rŒr­Údo_check_simulationrrÑrUrUrVrKÂs,rKz/pmr_latest_exposurecCs°t ¡}|r d|vr zptj|dddid}|jdkrxz$| ¡dddd dd }Wnd }Yn0t|d WS| ¡WSWq¬td ddYq¬0n td dddS)NZ workspace_urlÚAcceptz$application/vnd.physiome.pmr2.json.1r~rÓÚ collectionÚitemsrZlinksZhrefrPrrÌzInvalid workspace URLrÎzMissing workspace URL)rr•rmrˆrÖrŒrr)r…rNr‚rUrUrVÚpmr_latest_exposureÌs$  ÿ $ ÿrÖz/onto_term_lookupc CsÂtj d¡}ddi}dtji}t|ƒ}zbtjtj›d|||d}| ¡}|dd}|dd}|d kr||d }|d } nd d i} | WSt y¸} zt   d| ¡WYd} ~ n d} ~ 00t dƒS)NrErÓrƒr>r?)r‹rÑrŒrmÚtotalržrÚ_sourceÚlabelz not foundz.An error occured while fetching from SciCrunchr¼) rrÇrˆr<rr.rmZSCI_CRUNCH_INTERLEX_HOSTrŒrrrprgr) rEr‹rÑr$rÅr:rmr×r<rbrórUrUrVÚfind_by_onto_termæs& ÿÿ   "rÚz/search-readme/c Csxd|}ddtji}ztj||d}| ¡WStjjyr}z(t |¡t |ƒddœdfWYd}~Sd}~00dS)Nz2https://dash.readme.com/api/v1/docs/search?search=rŸzBasic r¢z9Readme is not currently reachable, please try again laterrFrÕ) r<ZREADME_API_KEYrmrrŒrnrorprgrh)r$r‚r‹rÅrsrUrUrVÚ search_readmesþ  þýrÛz/metricscCstSrQ)r–rUrUrUrVÚmetricssrÜz/event_updatedcCsrtj d¡}|tjkr*tdd|›dnDt ¡}|rbz t|ƒWStdd|›dYqn0n tddddS)NZevent_updated_secret_keyr zInvalid secret key: rÎrÌzInvalid event data: zMissing event data)rr‹rˆr<ZCTF_CDA_ACCESS_TOKENrr•r)Z secret_keyr¡rUrUrVÚ event_updateds   rÝz/all_dataset_idscCs"tƒ}dd„|Dƒ}d}| |¡S)NcSsg|] }t|ƒ‘qSrU)rh)ruÚelementrUrUrVry0r¦z#all_dataset_ids..z, )r Újoin)ÚlistZ string_listÚ delimiterrUrUrVÚall_dataset_ids-srâ)rAr)F)ñÚatexitr×Zapp.metrics.pennsieverZapp.metrics.contentfulrrrZ!scripts.update_contentful_entriesrrZapp.metrics.algoliarr r r Zapp.metrics.gar r Zscripts.monthly_statsrZ"scripts.update_featured_dataset_idrrZapp.osparc.servicesrrZboto3rŒrprÊrmÚ urllib.parserZ!apscheduler.schedulers.backgroundrZapscheduler.triggers.combiningrZapscheduler.triggers.daterZapscheduler.triggers.intervalrZbotocore.exceptionsrrrZflaskrrrrZ flask_corsrZflask_marshmallowrZ pennsiever Zpennsieve.baser!rqZPILr"Ú requests.authr#Z flask_cachingr$Zapp.scicrunch_requestsr%r&r'r(r)r*r+r,r-r.r/r0r1r2r3Zscripts.email_senderr4r5r6r7r8r9Ú threadingr:Z xml.etreer;Ú app.configr<Z app.dbtabler=r>r?Zapp.scicrunch_process_resultsr@rArBrCrDrErFZapp.serializerrGZ app.utilitiesrHrIZapp.osparc.osparcrJrÐrKrÒZapp.biolucida_process_resultsrŠrLr…Ú basicConfigrar”Z DEPLOY_ENVrMÚcacheZmar·rlÚclientZSPARC_PORTAL_AWS_KEYZSPARC_PORTAL_AWS_SECRETrNrZ DATABASE_URLr˜ÚAttributeErrorr›r¤ÚobjectrOZ errorhandlerrjZbefore_first_requestrtrr™rªr©r¨rr‡r‘Zmonthly_stats_email_schedulerÚmsZmonthly_stats_required_checkZadd_jobrTrŽr”r–r•r—r˜ršr›rœZfeatured_dataset_id_triggerr«ÚregisterZrouter®r»rÆZDEFAULT_S3_BUCKET_NAMErÉrÜrörÿrrrrrrr!r%r'r)r+r-r1r3r4r8r=r"rDrKrMrOrZrrcrdrerjrlrorqÚcachedrtr|r}r‚r†r‡rŒrŽr€r–r—r™ršrœrr´rÀrÂrÃrÅrÉrÍrÎrÖrÚrÛrÜrÝrârUrUrUrVÚsô                D    $    ü         ÿ         >  '           ,   0                     8 #