From 9968b41438a3c735dfd3832693ad76f4c4c32f8e Mon Sep 17 00:00:00 2001 From: PubMatic-OpenWrap Date: Wed, 23 Dec 2020 16:57:52 +0530 Subject: [PATCH] Prebid server version 0.138.0 Upgrade (#97) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * updating default hard-coded list of certs (#1201) Co-authored-by: Shalmali Patil * add admixer adapter (#1195) * Adding copying of gdpr consent string to openrtb bid request (#1189) * Adding copying of gdpr consent string to openrtb bid request * Updated video request to use OpenRTB Video and User objects * Fixing unit test failure message * Updates from code review comments * Updating unit test initialization * Updated mimes array construction * fix conversant sync pixel (#1208) * openx adapter: forward bid response currency in openx adapter if set (#1211) it was always set to the default USD before * add ucfunnel adapter (#1192) * Update required params for TheMediaGrid adapter (#1188) * add zeroclickfraud adapter (#1207) * add zeroclickfraud adapter * fixes for PR * fix casing of Zeroclickfraud * Fix Adform's parameters regex (#1214) * Added adform info file * Added Adform adapter and bidder * Updates from master * Removed usersyncInfo from Adform adapter. Inverted Imp type check. * Removed excessive loop * Updated with the last master * Create readme file for adform * Fix Adform's parameters regex Motivation: catastrophic backtracking during regex execution Details: - https://regex101.com/r/NNQrWq/1 - string to check "url_domain:keskustelu.suomi24.fi,url_path:/matkailu/matkakohteet/aasia,layout:lg,categories:Matkailu,main_category:Matkailu" Co-authored-by: v.statkevich Co-authored-by: Olga Linkevich * If Device.UA is not present in request body, init it with user-agent from header (#1219) * If Device.UA is not present in request body, init it with user-agent from request header if it's present * Moved User-Agent handler to parseVideoRequest func and added unit test * Minor clean up Co-authored-by: Veronika Solovei * Queued request timeout (#1217) Co-authored-by: Veronika Solovei * docs: adding currency support section (#1199) * Add ValueImpression Adapter (#1204) * Kidoz adapter (#1210) Co-authored-by: Ryan Haksi * Update auction.md (#1224) Fix type * Update auction.md (#1225) Fix typo. * Added logging to cache for video endpoint (#1220) * WIP added logging to cache for video endpoint * Updating cache call to use TTL from config * Updates from initial feedback * Log now includes HTTP headers * Fixed caching to use a new cache entry rather than appending to the VAST * Added feature where is query is set, the test flag is set in the request * Updated recorded response and handleError * Updates from code review comments * Changed recorded output to be only the debug ext * Removed extra marhal calls * Changed cache to be an endpoint dependency * Added debugLog struct to hold all debug related info * Numerous smaller changes * Further code cleanup and added unit tests for debug changes * Added missing error checks * Added unit test for error case * added VISX vendor ID for usersyncing (#1229) Co-authored-by: Aadesh Patel * First pass at phase 1 TCF 2.0 support (#1228) * First pass at phase 1 TCF 2.0 support * minor fixes * Update go-gdpr library and fix stuff * Fixes for PR comments * Updated price granularity unmarshal to accept empty values and ranges (#1230) * Update vendorID for TheMediaGrid s2s Bid Adapter (#1232) * treat 204 from FAN as a no bids response (#1233) Co-authored-by: Aadesh Patel * AMP CCPA Fix (#1187) * Update rubicon.md (#1234) * adding schain interface (#1203) * added Rewarded Video section (#1200) also edited all examples so they include the full openRTB context * nanointeractive adapter (#1213) * nanointeractive adapter * nanointeractive adapter, changes after review * nanointeractive adapter * nanointeractive adapter, changes after review * formatting * Typos Fix (#1236) * Fix Typo * Fixed More Typos * Moved hb_pc_cat_dur modification to be before caching (#1250) * replacing info@prebid.org maintainer email addrs (#1256) * aligning maintainer info (#1258) * Add kidoz bidder info (#1257) got this info from email communication with kidoz * Add Cropping of BAdv for Rubicon Adapter (#1254) * Add Cropping of BAdv for Rubicon Adapter BAdv size is limited to 50 * Fix after review Co-authored-by: Harbar Dmytro * Added metrics support to endpoint aspect (#1226) Co-authored-by: Veronika Solovei * Prebid Server adapter for Telaria (#1231) * TELARIA adapter. First Pass * Some refactoring * added the json files * fixed some tests and added the bidder info * fixed some tests and added the bidder info * added default user sync ur; * - Handling gzipped responses from our server * - more refactoring. * added the proper user sync default URL * changed the urls from dev to prod * changed up the required fields. Now AdCode in the Imp.Ext isn't required but Bid.SeatCode is required * change in the return type after decompressing * some refactoring * change in our config url * using pbs.yml to switch between our production and test URLs * setting default endpoint * - fixed the issue that was preventing telaria test cases to run. - added more test cases * - Modifications as per the changes requested by the maintainers. * Moved the seat code to imp.ext * Moved the seat code to imp.ext * Added 'Telaria: ' prefix for error messages * - Fixes for race conditions. Was modifying the original request object instead of a copy * cosmetic changes. * added params_test.go Co-authored-by: Vinay Prasad * #615 Beachfront URLs from config (#1238) * Add nil check errors when setting native asset types (#1260) * Bugfix: no bids from bidder handling (#1252) Co-authored-by: Veronika Solovei * Add missing categories to AppNexus -> IAB mapping file. (#1264) * Add missing categories to AppNexus -> IAB mapping file. * Remove entry for category 38 which was set to a primary IAB category instead of a sub-category. * Fix order of category 22 * Yieldone s2s Bid Adapter (#1242) * Added new Yieldone Bid s2s Adapter * Update endpoint for yieldone bid adapter * Fixes after review for Yieldone Bid s2s Adapter * Fix typeo in Yieldone s2s Bid Adapter * Fix: URL de sync (#1261) * populate the app ID in the FAN timeout notif url with the publisher ID (#1265) and the auction with the request ID Co-authored-by: Aadesh Patel * Added header User Agent decoding (#1268) * Added header User Agent decoding * Added header User Agent decoding: unit tests * Added header User Agent decoding: unit tests * Added check UA is encoded to avoid `+` converted to space Co-authored-by: Veronika Solovei * Ad Generation Adapter Integration. (#1253) * AdGeneration Integration. * update AdGeneration adapter. fix: some methods of the adgAdapter replace to functions. fix: unmarshal functions return a pointer. fix: header is defined once. fix: return when imps is appended * update AdGeneration Adapter. add: Added a comment in usersync. add: Added a test for parameters whose ID does not exist in params_test. change: Change to query creation by net/url. Added getRawQuery Test. fix: Changed variable names related to bidRequest. * Fix Go 1.14 Error Message Changes (#1271) * NinthDecimal Adapter (#1249) Co-authored-by: Chandra Prakash * * Add PubMatic bidder doc file (#1255) * Add app video capability to PubMatic bidder info file * Appnexus adapter: Add category mapping for government. (#1278) * Update a Freewheel mapping to Gaming category. (#1280) * Add AJA adapter (#1269) * OpenX adapter: Pass gdpr and gdpr_consent to user sync endpoint (#1282) I've also updated the test to avoid any confusion. * OpenX adapter: Enable video for app (#1281) * fix conversant sync pixel (#1284) * Add AdOcean adapter (#1273) * [ADOCEAN-20132] AdOcean adapter * [ADOCEAN-20132] AdOcean adapter - support for gdpr * [ADOCEAN-20132] AdOcean adapter - tests * [ADOCEAN-20132] AdOcean adapter - user sync * [ADOCEAN-20132] AdOcean adapter - formatting * [ADOCEAN-20132] AdOcean adapter - send uuid to emitter * [ADOCEAN-20132] adocean adapter - return nil if there is no creative * [ADOCEAN-20132] AdOcean adapter - add version parameter * [ADOCEAN-20132] AdOcean adapter - optimization * [ADOCEAN-20132] AdOcean adapter - add to syncer_test.go * [ADOCEAN-20132] AdOcean adapter - changes after review: * remove whitespaces in js code on adapter initialization instead on every request * check if request.Site is not nil * reuse newUri variable * [ADOCEAN-20132] AdOcean adapter - changes after review: * do not terminate the auction on a single faulty bid * [ADOCEAN-20132] AdOcean adapter - changes after review: * remove unnecessary input parameters check * small optimization * LunaMedia Adapter (#1285) Co-authored-by: Chandra Prakash * [Sharethrough] Add CCPA support (#1263) * Handle gzip responses from ad server correctly * Bump to version 8 * [Go Modules] Add proxy (#1079) * Add SSL cert for accessing stored request API (#1087) * [misspell] fix a misspell (#1102) * update static bidder params for rubicon video to follow the json marshalling names (#1100) * Switching yieldmo auction endpoint from http to https (#1103) * Add Datablocks Adapter (#1095) * datablocks bid adapter * ttx * add test json * add coverage * redo ttx * formatted * better error handling * additional tests and recomended fixes * Adding translatecategories flag to includebrandcategory (#1098) * Making IAB category translation optional with translatecategories boolean in request * Updating exchange unit tests to remove extra bids * Updates from code review comments * Removed comment about default TranslateCategories value * Changed translateCat to translateCategories in tests * Combined helper functions in exchange_test related to TranslateCategories * Bid floor (#1085) * Currency handling fix (#1097) * facebook adapter refactor (#1064) * Kubient adapter (#1094) * [synacormedia] Update user sync url to be https (#1115) This detail was missed while setting up the adapter, but we would like to use https for the user sync. * Remove Go 1.11 Build Target (#1109) * Set "Secure" on Same SIte cookies (#1119) * TripleliftNative Adapter (#1114) * ignore swp files * start small * start really small * add a user sync * justify * triplelift adapter * add our endpoint * fix syntax * config stuff * compiler fixes * more config * add params * making progress * make our ext more exty * start making responses * more logic * fix compilation errors * can we just nil this out? * augment our json * radically simplify our json * fix errs * infer the bid type * fix syntax * fix comilation errors * rename * fix compilation error * config stuff * simplify params * more config stuff * fixes * revert this * fix up the extension * getting closer * add a test * update config * update bidder params * add the floor here, too * add a usersync test * validation, ws, and a test * update tests * fix test * update email * why not * change email * preprocess requests * do some parsing * take care of some errors * floor is optional * ws * remove native * everything is either banner or video * this should be a float * floor to floor * fix compilation errors * add some tests * more tests * more tests * simplify * more progress * format * ws * rm * don't need this * fix test * fix test * don't ignore swap * change line back * report an error if there are no valid impressions for triplelift * check for either a Banner or Video object on the impression * more tests * mv * more tests * update triplelift end point * send native * ws * start changing tests * fix more tests * update config * add redirect to triplelift usersync * fix supplier id in triplelift_test * update tl usersync endpoint and test * fix tl supplier id in test json * update usersync test template * adjust inconsistency with test and sync url * mv * update packages * mv * mv * update * fix compilation errors * rename * rename some stuff * rename * rename * fix some compilation errors * ws * ws * add the extra info * add some extra info * add some files back * ws and such * updates * ws * fix compilation error * mv * rename * Revert "rename" This reverts commit 1b77c72e1eeee580148540fbdd880e70bf699709. * Revert "mv" This reverts commit 52a134ddfaf531fe6235e4751935d4266a36e78f. * it builds * cp a file * cp another file * fix a test * fix test * add the extra info * ws * add some logic * edit comment * it compiles * this is now public * call this * add the function * return nil * seems to be working * ws * seems to be working * ws * mv * starting to work * ws * add a new function * ws * fix tests * bug fix * update some stuff * revert * take out prints * fix up diff * fix up diff * update ws * fix * ws * omit the triplelift endppint * Revert "omit the triplelift endppint" This reverts commit 7abc3e46f0fbba39041da6fff7bb2335adc1fece. * populate the endpoint through the extinfo * ws * set disabled to be default * ws * update types * fixing tests * making progres * fix tests * fix tests * more fixes for tests * fixed tests * just use a comment * get rid of endpoint * restore endpoint * add some errors around unmarshalling * ws * ws * use the literal * ws * ws * update json * simplify * ws * restore tests * fail fast when grabbing invcode * use the right type * use a different error type * bump code coverage * add a new test * change error type * ws * break out test into its own function * JSON block that has a full data-center specific URL cache info (#1104) * Update Dockerfile and Makefile (#1099) * Add option for running tests as part of the docker image building * Update Makefile - Add ability to execute adapter specific tests - Execute targets for "all" rather than just printing the target name and usage - Remove use of non-existing "install" target from .PHONY targets - Remove "build" as a dependency for "image" * enable app requests for audience network (#1122) * [docs] fix markdown title (#1124) * Prometheus Refactor (#1108) * update default sync url (#1127) * Update sync url for BidderGrid adapter (#1120) * [SonarCloud] Legacy auction endpoint (#1017) * [currency converter] allow to deduce reverse rate (#1126) This CL allows the currency rate currency to deduce a currency rate even if not directly defined in the table but the reverse rate is present. E.q. USD => EUR is 1.0897 EUR => USD is not set Old behavior when asking rate from EUR to USD will not be found, New behavior is using the known reverse rate to deduce the rate. Rate for 2 USD will be 2 * (1 / 1.0897) * Updated handleError arguments to be pointers for video endpoint (#1128) * Updated handleError arguments to be pointers for video endpoint * Removing unneeded pointer to http.ResponseWriter * Adding units test for update to handleError * Revert changes to GetExtCacheData() made in #1104 (#1130) (#1131) * Better native request validation (#1132) * require the caller to define native assets[...].ID (#1123) * require the caller to define native assets[...].ID * Update assets-with-partial-ids.json * CCPA Phase 1: AMP Endpoint (#1125) * facebook: removed Auth-Token from header (replaced by authentication_id in the request body) (#1113) * Setuid Fix (#1121) * Update http refresh to use url builder. Fixes #1065 (#1133) * Add mapping of user.ext.eids[] for LiveIntent in Rubicon bidder (#1089) * support facebook app_secret config param (#1139) * CCPA Phase 1: Cookie Sync (#1135) * null check banner.h (#1142) * Add Pubnative Adapter (#1134) * Adding the passing of CCPA value to the bid request for video endpoint (#1143) * first draft (#1137) * CCPA Phase 2: Enforcement (#1138) * Gamoshi Adapter: Update cookie sync (#1146) * Simplify static/bidder-params/triplelift_native.json (#1152) * Added US Privacy support in TheMediaGrid server adapter (#1147) * Add TheMediaGrid server adapter * Add video support in TheMediaGrid s2s adapter * Update sync url for TheMediaGrid s2s adapter * Added CCPA support for TheMediaGrid s2s adapter * Fix sync url for TheMediaGrid adapter * CCPA User Sync Updates (#1153) * Marsmedia - add new bidder (#1118) * Add Applogy adapter (#1151) * enforce video.size_id for video imps in rubicon adapter (#1101) * Updated PubMatic endpoint to use https (#1155) * Update Example AppNexus Placement ID (#1160) * Fix Currency Converter Doesn't Output CUR (#1154) * Add custom JSON req/resp data to the analytics logging… (#1145) * Add custom JSON req/resp data to the analytics logging for the /openrtb2/video endpoint. * Add calls in unit tests to cover logging and jsonify of video object. * CCPA User Sync URL Updates (#1157) * Fixes audienceNetwork adapter ignoring banner.format sizes. (#1164) * adding yieldmo vendor id to usersync (#1166) * Add SmartRTB adapter (#1071) * Added new adapter for CPMStar ad network banners and video (#1159) * Update the Conversant sync pixel (#1161) * Add imp.ext.is_rewarded_inventory flag for rewarded video in Rubicon (#1170) * [currencies] fix GetInfo() null ref issue (#1169) This CL fixes the null ref on `RateConverter.GetInfo()` when rates are nil. Issue: #1136 * Fix triplelift User Sync (#1173) * Enhance Message For Cache Errors (#1175) * Fix PubMatic Usersync URL (#1178) Co-authored-by: pm-isha-bharti * [Synacormedia] Add tagId bidder parameter (#1165) * Remove all non-secure calls from eplanning adapter (#1179) * Expose Cache HTTP Settings (#1184) * Adding bid rejection messages to debug response (#1181) * Adds timeout notifications for Facebook (#1182) * VIS.X: added app type support (#1194) * Add Adoppler bidder support. (#1186) * Add Adoppler bidder support. * Address code review comments. Use JSON-templates for testing. * Fix misprint; Add url.PathEscape call for adunit URL parameter. * Adding support for deal prefixes (#1183) * updating default hard-coded list of certs (#1201) Co-authored-by: Shalmali Patil * add admixer adapter (#1195) * Adding copying of gdpr consent string to openrtb bid request (#1189) * Adding copying of gdpr consent string to openrtb bid request * Updated video request to use OpenRTB Video and User objects * Fixing unit test failure message * Updates from code review comments * Updating unit test initialization * Updated mimes array construction * fix conversant sync pixel (#1208) * openx adapter: forward bid response currency in openx adapter if set (#1211) it was always set to the default USD before * add ucfunnel adapter (#1192) * Update required params for TheMediaGrid adapter (#1188) * add zeroclickfraud adapter (#1207) * add zeroclickfraud adapter * fixes for PR * fix casing of Zeroclickfraud * Fix Adform's parameters regex (#1214) * Added adform info file * Added Adform adapter and bidder * Updates from master * Removed usersyncInfo from Adform adapter. Inverted Imp type check. * Removed excessive loop * Updated with the last master * Create readme file for adform * Fix Adform's parameters regex Motivation: catastrophic backtracking during regex execution Details: - https://regex101.com/r/NNQrWq/1 - string to check "url_domain:keskustelu.suomi24.fi,url_path:/matkailu/matkakohteet/aasia,layout:lg,categories:Matkailu,main_category:Matkailu" Co-authored-by: v.statkevich Co-authored-by: Olga Linkevich * If Device.UA is not present in request body, init it with user-agent from header (#1219) * If Device.UA is not present in request body, init it with user-agent from request header if it's present * Moved User-Agent handler to parseVideoRequest func and added unit test * Minor clean up Co-authored-by: Veronika Solovei * Queued request timeout (#1217) Co-authored-by: Veronika Solovei * docs: adding currency support section (#1199) * Add ValueImpression Adapter (#1204) * Kidoz adapter (#1210) Co-authored-by: Ryan Haksi * Update auction.md (#1224) Fix type * Update auction.md (#1225) Fix typo. * Added logging to cache for video endpoint (#1220) * WIP added logging to cache for video endpoint * Updating cache call to use TTL from config * Updates from initial feedback * Log now includes HTTP headers * Fixed caching to use a new cache entry rather than appending to the VAST * Added feature where is query is set, the test flag is set in the request * Updated recorded response and handleError * Updates from code review comments * Changed recorded output to be only the debug ext * Removed extra marhal calls * Changed cache to be an endpoint dependency * Added debugLog struct to hold all debug related info * Numerous smaller changes * Further code cleanup and added unit tests for debug changes * Added missing error checks * Added unit test for error case * added VISX vendor ID for usersyncing (#1229) Co-authored-by: Aadesh Patel * First pass at phase 1 TCF 2.0 support (#1228) * First pass at phase 1 TCF 2.0 support * minor fixes * Update go-gdpr library and fix stuff * Fixes for PR comments * Updated price granularity unmarshal to accept empty values and ranges (#1230) * Update vendorID for TheMediaGrid s2s Bid Adapter (#1232) * treat 204 from FAN as a no bids response (#1233) Co-authored-by: Aadesh Patel * AMP CCPA Fix (#1187) * Update rubicon.md (#1234) * adding schain interface (#1203) * added Rewarded Video section (#1200) also edited all examples so they include the full openRTB context * nanointeractive adapter (#1213) * nanointeractive adapter * nanointeractive adapter, changes after review * nanointeractive adapter * nanointeractive adapter, changes after review * formatting * Typos Fix (#1236) * Fix Typo * Fixed More Typos * Moved hb_pc_cat_dur modification to be before caching (#1250) * Handle CCPA + enable gzip response [#169984259] * Addressing review (#273) [#169984259] * Remove custom gzip logic (#280) * Getting rid of custom gzip logic [#169984259] * Restore prod ad server url [#169984259] Co-authored-by: Benjamin Co-authored-by: guscarreon Co-authored-by: Aadesh Co-authored-by: Winston-Yieldmo <46379634+Winston-Yieldmo@users.noreply.github.com> Co-authored-by: htang555 Co-authored-by: Cameron Rice <37162584+camrice@users.noreply.github.com> Co-authored-by: ah-tappx <46002207+ah-tappx@users.noreply.github.com> Co-authored-by: hhhjort <31041505+hhhjort@users.noreply.github.com> Co-authored-by: Marsel Co-authored-by: Corey Kress Co-authored-by: Scott Kay Co-authored-by: Kevin Kerr Co-authored-by: Mansi Nahar Co-authored-by: Benjamin Co-authored-by: TheMediaGrid <44166371+TheMediaGrid@users.noreply.github.com> Co-authored-by: Austin Bischoff Co-authored-by: rpanchyk Co-authored-by: Florian Hartwig Co-authored-by: Salomon Rada Co-authored-by: vladi-mmg Co-authored-by: Aleksei Lin Co-authored-by: PubMatic-OpenWrap Co-authored-by: jmaynardxandr <46759873+jmaynardxandr@users.noreply.github.com> Co-authored-by: evanmsmrtb Co-authored-by: CPMStar Co-authored-by: johnwier <49074029+johnwier@users.noreply.github.com> Co-authored-by: pm-isha-bharti Co-authored-by: Seba Perez Co-authored-by: Michael Kuryshev Co-authored-by: Viacheslav Chimishuk Co-authored-by: Shalmali Patil Co-authored-by: DmitryStashkevich <34479135+DmitryStashkevich@users.noreply.github.com> Co-authored-by: vstatkevich Co-authored-by: v.statkevich Co-authored-by: Olga Linkevich Co-authored-by: Veronika Solovei Co-authored-by: Veronika Solovei Co-authored-by: bretg Co-authored-by: thuyhq <61451682+thuyhq@users.noreply.github.com> Co-authored-by: rhaksi-kidoz <61601767+rhaksi-kidoz@users.noreply.github.com> Co-authored-by: Ryan Haksi Co-authored-by: ACannuniRP <57228257+ACannuniRP@users.noreply.github.com> Co-authored-by: Aadesh Patel Co-authored-by: Rade Popovic <32302052+nanointeractive@users.noreply.github.com> * Remove Outdated GDPR AMP Special Case (#1283) * Stricter Privacy Scrubbing (#1286) * Stricter Privacy Scrubbing * Update Unit Test Style * Fixed Whitespace * Add Adapter Orbidder (#1275) Co-authored-by: Volk, Rainer Co-authored-by: RainerVolk4014 <53347752+RainerVolk4014@users.noreply.github.com> Co-authored-by: rvolk <> Co-authored-by: Hendrik Iseke Co-authored-by: hendrikiseke1979 <53309111+hendrikiseke1979@users.noreply.github.com> * Added OpenX Bidder adapter documentation (#1291) * OpenX adapter: Pass rewarded video flag (#1290) * Bugfix for missing fields in imp.video (#1297) Co-authored-by: Veronika Solovei * Add cpmOverride (#1289) * Add cpmOverride Enabled `request.ext.rubicon.debug.cpmOverride` and `request.imp[].ext.rubicon.debug.cpmOverride` processing. Updates tests * Remove unnecessary error checks and add shallow copy * Fixed same pointer * Add Beintoo adapter (#1274) * Add Beintoo adapter * Yeahmobi adapter (#1279) Co-authored-by: junping.zhao * advangelists: Vendor id update (#1307) Co-authored-by: Chandra Prakash * Consumable: Support GDPR and US Privacy consent (#1300) * Restore the AMP privacy exception as an option. (#1311) * Restore the AMP privacy exception as an option. * Adds missing test case * More PR feedback * Remove unused constant * Comment tweak * consumable: Correct GDPR vendor ID to 591. (#1309) fixes #1299. * VIS.X: fix bid.ID, bid.CrID and set default currency value (#1296) * Fix debug log error messages (#1270) * Fixing missing error messages for debug logging * Updated formatting of debug log message * Updated unit tests for debug log to have test flag enabled * Cleaned up debug log implementation * Updates from review comments * Cleaned up field and function names * Added replacer for <> characters * Added cache string unit test * Moved regex from function to struct field * Moved debug regex to endpoint deps * Moving regex initialization to NewVideoEndpoint * MobileFuse Adapter (#1303) Co-authored-by: Dan Barnett * eplanning: Support for apps (#1306) * Introduce Adhese adapter (#1292) Co-authored-by: Mateusz * privacy: Potential JSON injection (#1304) * Updating bidder params for Advangelists (#1316) * Updating placement info on bidder params Co-authored-by: Chandra Prakash * Change placement of cpmoverride for Rubicon (#1310) * increasing the stale period to 2 months (#1305) * Add Go 1.14 Build Target (#1314) * Privacy: Remove user.ext.eids (#1294) * Privacy: Remove user.ext.eids * Extract To A Method * Minor Refactor + More Tests * Performance Tweak * Removed some redundant methods (#1320) * TELARIA adapter. First Pass * Some refactoring * added the json files * fixed some tests and added the bidder info * fixed some tests and added the bidder info * added default user sync ur; * - Handling gzipped responses from our server * - more refactoring. * added the proper user sync default URL * changed the urls from dev to prod * changed up the required fields. Now AdCode in the Imp.Ext isn't required but Bid.SeatCode is required * change in the return type after decompressing * some refactoring * change in our config url * using pbs.yml to switch between our production and test URLs * setting default endpoint * - fixed the issue that was preventing telaria test cases to run. - added more test cases * - Modifications as per the changes requested by the maintainers. * Moved the seat code to imp.ext * Moved the seat code to imp.ext * Added 'Telaria: ' prefix for error messages * - Fixes for race conditions. Was modifying the original request object instead of a copy * cosmetic changes. * added params_test.go * Removed some redundant methods. * Removed a comment Co-authored-by: Vinay Prasad * Beachfront: GDPR id (issue 1301) and documentation updates (#1321) * Defined cookie sync URL in config, cleared deprecated comment in usersync * Update beachfront.md * editing documentation * updated gdpr id - issue 1301 * Add Yieldlab Adapter (#1287) Co-authored-by: Mirko Feddern Signed-off-by: Alex Klinkert Co-authored-by: Alexander Pinnecke Co-authored-by: Alex Klinkert Co-authored-by: Mirko Feddern * Update adtelligent ortb endpoint (#1318) * Change on eplanning endpoint (#1327) * Enable full TCF2 support (#1302) * New config options * Enble TCF2 fields and logic * Resolves some PR comments * More tests * gofmt * Added enforcement tests for split GDPR/GDPRGeo * Testing tweaks * No longer ignore enforce purpose 1 on allowSync() * Removes Purpose 4 * Change on eplanning endpoint (hostname) (#1328) * Districtm Dmx: new adapter (#1209) Co-authored-by: steve-a-districtm * Fix sync url for Yieldone s2s Bid Adapter (#1336) * Fix typo in Yieldone sync url * CCPA Video Bug (#1333) * Add Pubnative bidder documentation (#1340) * Timeout notification monitoring and debugging (#1322) * Add Adtarget server adapter (#1319) * Add Adtarget server adapter * Suggested changes for Adtarget * Update Auction OpenRTB Sample (#1342) * Update Auction OpenRTB Sample * Removed Extra "Or" * Triplelift: Add SRA Support (#1347) * Privacy: Limit Ad Tracking (#1334) * Avoid overriding AMP request original size with mutli-size (#1352) * Extra logging for timeout notifications (#1349) * Consumable: Correct bid type, should always be "banner". (#1359) * Build With Go 1.14 (#1350) * Category mapping changes from product team. (#1348) * Adds Avocet adapter (#1354) * AdOcean adapter - Support for sizes defined in prebid configuration. (#1339) support for multiple sizes bump version to 1.1.0 * Log account id and all bidder names when recovering from OpenRTB auction bidder… (#1358) * Adding Smartadserver adapter (#1346) Co-authored-by: tadam * Added additional Ext Param (#1357) Co-authored-by: Vinay Prasad * Adman adapter (#1356) Co-authored-by: Aiholkin * PBS-632 add max connections per host config setting to general http a… (#1366) * Add ext.bidder.zoneid for Kubient adapater (#1367) * Add ext.bidder.zoneid for Kubient adapater * Check the number of Imps. zoneid is optional. * Improved IPv6 Support + Private Network Filtering (#1362) * Change endpont address (#1370) * Adman adapter * add adman line to syner test * add tests * fix issues * fix web banner test * add 404 banner * fmt * rase coverage * del redundant files * change endpont address * change config endpoint Co-authored-by: Aiholkin * Don't override test parameter (#1373) * OpenX + Facebook Hardening (#1368) * Updating Conversant endpoint url (#1376) * Metrics for TCF 2 adoption (#1360) * Fall back to constant rates when the currency rates endpoint i… (#1364) * TheMediaGrid: added app type support (#1377) * user.ext.eids support in adform adapter (#1381) * Add Logicad adapter (#1382) * Fix Previous Merge Conflict (#1392) * Kubient: Change default endpont address (#1398) * Add support for multiple root schain nodes (#1374) * Update endpoint for latest release by districtm (#1401) Co-authored-by: steve-a-districtm * Set OpenRTB DNT From HTTP Header (#1397) * Add video for InApp support (#1399) * Timeout fix (#1390) * Privacy Request Metrics (#1400) * Privacy Request Metrics * Fix Bug + Add Unit Tests * Fixed Tests * Fix Typo * Parse Site.Publisher.ID from Amp Auction HTTP Req Query Parameter "account" (#1403) * Facebook Only Supports App Impressions (#1396) * fix: Change currency of ad-generation's bidResponse according to bidRequest (#1383) * Adding primary categories to freewheel mapping (#1407) * Add Outgoing Connection Metrics (#1343) * Pubmatic: Support for video duration and primary category (#1384) * Adding suport for video duration and primary category in pubmatic adapter * Adding code review changes for PR-1384 * Adding changes for syntaxNode suggestion Co-authored-by: Isha Bharti * Add IPv6 Non-Public Network (#1417) * GumGum: adds support for video (#1408) * OpenX adapter: pass optional platform (PBID-598) (#1421) * Adds keyvalue hb_format support (#1414) * feat: Add new logger module - Pubstack Analytics Module (#1331) * Pubstack Analytics V1 (#11) * V1 Pubstack (#7) * feat: Add Pubstack Logger (#6) * first version of pubstack analytics * bypass viperconfig * commit #1 * gofmt * update configuration and make the tests pass * add readme on how to configure the adapter and update the network calls * update logging and fix intake url definition * feat: Pubstack Analytics Connector * fixing go mod * fix: bad behaviour on appending path to auction url * add buffering * support bootstyrap like configuration * implement route for all the objects * supports termination signal handling for goroutines * move readme to the correct location * wording * enable configuration reload + add tests * fix logs messages * fix tests * fix log line * conclude merge * merge * update go mod Co-authored-by: Amaury Ravanel * fix duplicated channel keys Co-authored-by: Amaury Ravanel * first pass - PR reviews * rename channel* -> eventChannel * dead code * Review (#10) * use json.Decoder * update documentation * use nil instead []byte("") * clean code * do not use http.DefaultClient * fix race condition (need validation) * separate the sender and buffer logics * refactor the default configuration * remove error counter * Review GP + AR * updating default config * add more logs * remove alias fields in json * fix json serializer * close event channels Co-authored-by: Amaury Ravanel * fix race condition * first pass (pr reviews) * refactor: store enabled modules into a dedicated struct * stop goroutine * test: improve coverage * PR Review * Revert "refactor: store enabled modules into a dedicated struct" This reverts commit f57d9d61680c74244effc39a5d96d6cbb2f19f7d. # Conflicts: # analytics/config/config_test.go Co-authored-by: Amaury Ravanel * New bid adapter for Smaato (#1413) Co-authored-by: vikram Co-authored-by: Stephan * New Adprime adapter (#1418) Co-authored-by: Aiholkin * Separate "debug" behavior from "billable" behavior (#1387) * Remove redundad struct (#1432) * Tcf2 id support (#1420) * Default TCF1 GVL in anticipation of IAB no longer hosting the v1 GVL (#1433) * update to the latest go-gdpr release (#1436) * Video endpoint bid selection enhancements (#1419) Co-authored-by: Veronika Solovei * [WIP] Bid deduplication enhancement (#1430) Co-authored-by: Veronika Solovei * Refactor rate converter separating scheduler from converter logic to improve testability (#1394) * Fix TCF1 Fetcher Fallback (#1438) * Eplanning adapter: Get domain from page (#1434) * Fix no bid debug log (#1375) * Update the fallback GVL to last version (#1440) * Enable geo activation of GDPR flag (#1427) * Validate External Cache Host (#1422) * first draft * Little tweaks * Scott's review part 1 * Scott's review corrections part 2 * Scotts refactor * correction in config_test.go * Correction and refactor * Multiple return statements * Test case refactor Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon * Fixes bug (#1448) * Fixes bug * shortens list * Added adpod_id to request extension (#1444) * Added adpod_id to request -> ext -> appnexus and modified requests splitting based on pod * Unit test fix * Unit test fix * Minor unit test fixes * Code refactoring * Minor code and unit tests refactoring * Unit tests refactoring Co-authored-by: Veronika Solovei * Adform adapter: additional targeting params added (#1424) * Fix minor error message spelling mistake "vastml" -> "vastxml" (#1455) * Fixing comment for usage of deal priority field (#1451) * moving docs to website repo (#1443) * Fix bid dedup (#1456) Co-authored-by: Veronika Solovei * consumable: Correct width and height reported in response. (#1459) Prebid Server now responds with the width and height specified in the Bid Response from Consumable. Previously it would reuse the width and height specified in the Bid Request. That older behaviour was ported from an older version of the prebid.js adapter but is no longer valid. * Panics happen when left with zero length []Imp (#1462) * Add Scheme Option To External Cache URL (#1460) * Update gamma adapter (#1447) * Gamma SSP Adapter * Add Gamma SSP server adapter * increase coverage * Fix conflict with base master * Add check MediaType for Imp * Implement Multi Imps request * Changes requested * remove bad-request * increase coverage * Remove duplicate test file * Update gamma.go * Update gamma.go * Update gamma.go * Update config.go Remove Gamma User Sync Url from config * Gamma SSP Adapter * Add Gamma SSP server adapter * increase coverage * Fix conflict with base master * Add check MediaType for Imp * Implement Multi Imps request * Changes requested * remove bad-request * increase coverage * Remove duplicate test file * Update gamma.go * Update gamma.go * update gamma adapter * return nil when have No-Bid Signaling * add missing-adm.json * discard the bid that's missing adm * discard the bid that's missing adm * escape vast instead of encoded it * expand test coverage Co-authored-by: Easy Life * fix: avoid unexpected EOF on gz writer (#1449) * Smaato adapter: support for video mediaType (#1463) Co-authored-by: vikram * Rubicon liveramp param (#1466) Add liveramp mapping to user.ext should translate the "liveramp.com" id from the "user.ext.eids" array to "user.ext.liveramp_idl" as follows: ``` { "user": { "ext": { "eids": [{ "source": 'liveramp.com', "uids": [{ "id": "T7JiRRvsRAmh88" }] }] } } } ``` to XAPI: ``` { "user": { "ext": { "liveramp_idl": "T7JiRRvsRAmh88" } } } ``` * Consolidate StoredRequest configs, add validation for all data types (#1453) * Fix Test TestEventChannel_OutputFormat (#1468) * Add ability to randomly generate source.TID if empty and set publisher.ID to resolved account ID (#1439) * Add support for Account configuration (PBID-727, #1395) (#1426) * Minor changes to accounts test coverage (#1475) * Brightroll adapter - adding config support (#1461) * Refactor TCF 1/2 Vendor List Fetcher Tests (#1441) * Add validation checker for PRs and merges with github actions (#1476) * Cache refactor (#1431) Reason: Cache has Fetcher-like functionality to handle both requests and imps at a time. Internally, it still uses two caches configured and searched separately, causing some code repetition. Reusing this code to cache other objects like accounts is not easy. Keeping the req/imp repetition in fetcher and out of cache allows for a reusable simpler cache, preserving existing fetcher functionality. Changes in this set: Cache is now a simple generic id->RawMessage store fetcherWithCache handles the separate req and imp caches ComposedCache handles single caches - but it does not appear to be used Removed cache overlap tests since they do not apply now Slightly less code * Pass Through First Party Context Data (#1479) * Added new size 640x360 (Id: 198) (#1490) * Refactor: move getAccount to accounts package (from openrtb2) (#1483) * Fixed TCF2 Geo Only Enforcement (#1492) * New colossus adapter [Clean branch] (#1495) Co-authored-by: Aiholkin * New: InMobi Prebid Server Adapter (#1489) * Adding InMobi adapter * code review feedback, also explicitly working with Imp[0], as we don't support multiple impressions * less tolerant bidder params due to sneaky 1.13 -> 1.14+ change * Revert "Added new size 640x360 (Id: 198) (#1490)" (#1501) This reverts commit fa23f5c226df99a9a4ef318100fdb7d84d3e40fa. * CCPA Publisher No Sale Relationships (#1465) * Fix Merge Conflict (#1502) * Update conversant adapter for new prebid-server interface (#1484) * Implement returnCreative (#1493) * Working solution * clean-up * Test copy/paste error Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon * ConnectAd S2S Adapter (#1505) * between adapter (#1437) Co-authored-by: Alexey Elymanov * Invibes adapter (#1469) Co-authored-by: aurel.vasile * Refactor postgres event producer so it will run either the full or de… (#1485) * Refactor postgres event producer so it will run either the full or delta query periodically * Minor cleanup, follow golang conventions, declare const slice, add test cases * Remove comments * Bidder Uniqueness Gatekeeping Test (#1506) * ucfunnel adapter update end point (#1511) * Refactor EEAC map to be more in line with the nonstandard publisher map (#1514) * Added bunch of new sizes (#1516) * New krushmedia bid adapter (#1504) * Invibes: Generic domainId parameter (#1512) * Smarty ads adapter (#1500) Co-authored-by: Kushneryk Pavlo Co-authored-by: user * Add vscode remote container development files (#1481) * First commit (#1510) Co-authored-by: Gus Carreon * Vtrack and event endpoints (#1467) * Rework pubstack module tests to remove race conditions (#1522) * Rework pubstack module tests to remove race conditions * PR feedback * Remove event count and add helper methods to assert events received on channel * Updating smartadserver endpoint configuration. (#1531) Co-authored-by: tadam * Add new size 500x1000 (ID: 548) (#1536) * Fix missing Request parameter for Adgeneration Adapter (#1525) * Fix endpoint url for TheMediaGrid Bid Adapter (#1541) * Add Account cache (#1519) * Add bidder name key support (#1496) * Simplifying exchange module: bidResponseExt gets built anyway (#1518) * first draft * Scott's feedback * stepping out real quick * add cache errors to bidResponseExt before marshalling * Removed vim's swp file Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon * Correct GetCpmStringValue's second return value (#1520) * Add metrics to capture stored data fetch all/delta durations with fetch status (#1515) * Adds preferDeals support (#1528) * Emxd 3336 add app video ctv (#1529) * Adapter changes for app and video support * adding ctv devicetype test case * Adding whitespace * Updates based on feedback from Prebid team * protocol bug fix and testing * Modifying test cases to accomodate new imp.ext field * bidtype bug fix and additonal testcase for storeUrl Co-authored-by: Rakesh Balakrishnan Co-authored-by: Dan Bogdan * Add http api for fetching accounts (#1545) * Add missing postgres cache init config validation * Acuity ads adapter (#1537) Co-authored-by: Kushneryk Pavlo * Yieldmo app support in yaml file (#1542) Co-authored-by: Winston * Add metrics for account cache (#1543) * [Invibes] remove user sync for invibes (#1550) * [invibes] new bidder stub * [invibes] make request * [invibes] bid request parameters * [invibes] fix errors, add tests * [invibes] new version of MakeBids * cleaning code * [invibes] production urls, isamp flag * [invibes] fix parameters * [invibes] new test parameter * [invibes] change maintainer email * [invibes] PR fixes * [invibes] fix parameters test * [invibes] refactor endpoint template and bidVersion * [Invibes] fix tests * [invibes] resolve PR * [invibes] fix test * [invibes] fix test * [invibes] generic domainId parameter * [invibes] remove invibes cookie sync * [Invibes] comment missing Usersync Co-authored-by: aurel.vasile * Add Support For imp.ext.prebid For DealTiers (#1539) * Add Support For imp.ext.prebid For DealTiers * Remove Normalization * Add Accounts to http cache events (#1553) * Fix JSON tests ignore expected message field (#1450) * NoBid version 1.0. Initial commit. (#1547) Co-authored-by: Reda Guermas * Added dealTierSatisfied parameters in exchange.pbsOrtbBid and openrtb_ext.ExtBidPrebid and dealPriority in openrtb_ext.ExtBidPrebid (#1558) Co-authored-by: Shriprasad * Add client/AccountID support into Adoppler adapter. (#1535) * Optionally read IFA value and add it the the request url (Adhese) (#1563) * Add AMX RTB adapter (#1549) * update Datablocks usersync.go (#1572) * 33Across: Add video support in adapter (#1557) * SilverMob adapter (#1561) * SilverMob adapter * Fixes andchanges according to notes in PR * Remaining fixes: multibids, expectedMakeRequestsErrors * removed log * removed log * Multi-bid test * Removed unnesesary block Co-authored-by: Anton Nikityuk * Updated ePlanning GVL ID (#1574) * update adpone google vendor id (#1577) * ADtelligent gvlid (#1581) * Add account/ host GDPR enabled flags & account per request type GDPR enabled flags (#1564) * Add account level request type specific and general GDPR enabled flags * Clean up test TestAccountLevelGDPREnabled * Add host-level GDPR enabled flag * Move account GDPR enable check as receiver method on accountGDPR * Remove mapstructure annotations on account structs * Minor test updates * Re-add mapstructure annotations on account structs * Change RequestType to IntegrationType and struct annotation formatting * Update comment * Update account IntegrationType comments * Remove extra space in config/accounts.go via gofmt * DMX Bidfloor fix (#1579) * adform bidder video bid response support (#1573) * Fix Beachfront JSON tests (#1578) * Add account CCPA enabled and per-request-type enabled flags (#1566) * Add account level request-type-specific and general CCPA enabled flags * Remove mapstructure annotations on CCPA account structs and clean up CCPA tests * Adjust account/host CCPA enabled flag logic to incorporate feedback on similar GDPR feature * Add shared privacy policy account integration data structure * Refactor EnabledForIntegrationType methods on account privacy objects * Minor test refactor * Simplify logic in EnabledForIntegrationType methods * Refactored HoldAuction Arguments (#1570) * Fix bug in request.imp.ext Validation (#1575) * First draft * Brian's reivew * Removed leftover comments Co-authored-by: Gus Carreon * Updating import statements for v0.138.0 upgrade * UOE-5690: Fixing merging issues * UOE-5690 Fixing merging issues * prebid-server v0.138 upgrade: fixing merging issue * Prebid-upgrade Fixing test cases * Prebid-server upgrade: removing unwanted files Co-authored-by: Shalmali Patil Co-authored-by: DmitryStashkevich <34479135+DmitryStashkevich@users.noreply.github.com> Co-authored-by: Cameron Rice <37162584+camrice@users.noreply.github.com> Co-authored-by: johnwier <49074029+johnwier@users.noreply.github.com> Co-authored-by: Scott Kay Co-authored-by: guscarreon Co-authored-by: TheMediaGrid <44166371+TheMediaGrid@users.noreply.github.com> Co-authored-by: htang555 Co-authored-by: vstatkevich Co-authored-by: v.statkevich Co-authored-by: Olga Linkevich Co-authored-by: Veronika Solovei Co-authored-by: Veronika Solovei Co-authored-by: bretg Co-authored-by: thuyhq <61451682+thuyhq@users.noreply.github.com> Co-authored-by: rhaksi-kidoz <61601767+rhaksi-kidoz@users.noreply.github.com> Co-authored-by: Ryan Haksi Co-authored-by: ACannuniRP <57228257+ACannuniRP@users.noreply.github.com> Co-authored-by: Aadesh Co-authored-by: Aadesh Patel Co-authored-by: hhhjort <31041505+hhhjort@users.noreply.github.com> Co-authored-by: Rade Popovic <32302052+nanointeractive@users.noreply.github.com> Co-authored-by: Dmitriy Co-authored-by: Harbar Dmytro Co-authored-by: Telaria Engineering <36203956+telariaEng@users.noreply.github.com> Co-authored-by: Vinay Prasad Co-authored-by: Krzysztof Desput Co-authored-by: Mansi Nahar Co-authored-by: jmaynardxandr <46759873+jmaynardxandr@users.noreply.github.com> Co-authored-by: hbanalytics <55453525+hbanalytics@users.noreply.github.com> Co-authored-by: chino117 Co-authored-by: Ad Generation Co-authored-by: trchandraprakash <47793448+trchandraprakash@users.noreply.github.com> Co-authored-by: Chandra Prakash Co-authored-by: Mike Chowla Co-authored-by: Taiki Sakamoto Co-authored-by: Laurentiu Badea Co-authored-by: Marcin Muras <47107445+mmuras@users.noreply.github.com> Co-authored-by: Mathieu Pheulpin Co-authored-by: Benjamin Co-authored-by: Winston-Yieldmo <46379634+Winston-Yieldmo@users.noreply.github.com> Co-authored-by: ah-tappx <46002207+ah-tappx@users.noreply.github.com> Co-authored-by: Marsel Co-authored-by: Corey Kress Co-authored-by: Kevin Kerr Co-authored-by: Benjamin Co-authored-by: Austin Bischoff Co-authored-by: rpanchyk Co-authored-by: Florian Hartwig Co-authored-by: Salomon Rada Co-authored-by: vladi-mmg Co-authored-by: Aleksei Lin Co-authored-by: evanmsmrtb Co-authored-by: CPMStar Co-authored-by: pm-isha-bharti Co-authored-by: Seba Perez Co-authored-by: Michael Kuryshev Co-authored-by: Viacheslav Chimishuk Co-authored-by: Arne Schulz Co-authored-by: Volk, Rainer Co-authored-by: RainerVolk4014 <53347752+RainerVolk4014@users.noreply.github.com> Co-authored-by: Hendrik Iseke Co-authored-by: hendrikiseke1979 <53309111+hendrikiseke1979@users.noreply.github.com> Co-authored-by: Jimmy Tu Co-authored-by: ddantuonobeintoo <58686785+ddantuonobeintoo@users.noreply.github.com> Co-authored-by: zhaojp <327199034@qq.com> Co-authored-by: junping.zhao Co-authored-by: Daniel Cassidy Co-authored-by: dtbarne <7635750+dtbarne@users.noreply.github.com> Co-authored-by: Dan Barnett Co-authored-by: Sander Co-authored-by: Mateusz Co-authored-by: Jim Naumann Co-authored-by: Mirko Feddern <3244291+mirkorean@users.noreply.github.com> Co-authored-by: Alexander Pinnecke Co-authored-by: Alex Klinkert Co-authored-by: Mirko Feddern Co-authored-by: Gena Co-authored-by: Steve Alliance Co-authored-by: steve-a-districtm Co-authored-by: Artur Aleksanyan Co-authored-by: Brandon Ling <51931757+blingster7@users.noreply.github.com> Co-authored-by: Richard Lee <14349+dlackty@users.noreply.github.com> Co-authored-by: Simon Critchley Co-authored-by: Brian Sardo <1168933+bsardo@users.noreply.github.com> Co-authored-by: tadam75 Co-authored-by: tadam Co-authored-by: SmartyAdman <59048845+SmartyAdman@users.noreply.github.com> Co-authored-by: Aiholkin Co-authored-by: Marsel Co-authored-by: AaronColbyPrice <67345931+AaronColbyPrice@users.noreply.github.com> Co-authored-by: Jurij Sinickij Co-authored-by: logicad Co-authored-by: Daniel Barrigas Co-authored-by: susyt Co-authored-by: gpolaert Co-authored-by: Amaury Ravanel Co-authored-by: Vikram Co-authored-by: vikram Co-authored-by: Stephan Co-authored-by: Adprime <64427228+Adprime@users.noreply.github.com> Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon Co-authored-by: Jurij Sinickij Co-authored-by: Rob Hazan Co-authored-by: GammaSSP <35954362+gammassp@users.noreply.github.com> Co-authored-by: Easy Life Co-authored-by: smithaammassamveettil <39389834+smithaammassamveettil@users.noreply.github.com> Co-authored-by: hdeodhar <35999856+hdeodhar@users.noreply.github.com> Co-authored-by: Bill Newman Co-authored-by: Daniel Lawrence Co-authored-by: rtuschkany <35923908+rtuschkany@users.noreply.github.com> Co-authored-by: Alexey Elymanov Co-authored-by: Alexey Elymanov Co-authored-by: invibes <51820283+invibes@users.noreply.github.com> Co-authored-by: aurel.vasile Co-authored-by: ucfunnel <39581136+ucfunnel@users.noreply.github.com> Co-authored-by: Krushmedia <71434282+Krushmedia@users.noreply.github.com> Co-authored-by: Kushneryk Pavel Co-authored-by: Kushneryk Pavlo Co-authored-by: user Co-authored-by: Gus Carreon Co-authored-by: Gus Carreon Co-authored-by: Dan Bogdan <43830380+EMXDigital@users.noreply.github.com> Co-authored-by: Rakesh Balakrishnan Co-authored-by: Dan Bogdan Co-authored-by: AcuityAdsIntegrations <72594990+AcuityAdsIntegrations@users.noreply.github.com> Co-authored-by: Winston Co-authored-by: redaguermas Co-authored-by: Reda Guermas Co-authored-by: ShriprasadM Co-authored-by: Shriprasad Co-authored-by: Nick Jacob Co-authored-by: Aparna Rao Co-authored-by: silvermob <73727464+silvermob@users.noreply.github.com> Co-authored-by: Anton Nikityuk Co-authored-by: Sergio --- .devcontainer/Dockerfile | 18 + .devcontainer/devcontainer.json | 45 + .github/workflows/validate.yml | 28 + .gitignore | 2 +- README.md | 19 +- account/account.go | 69 + account/account_test.go | 94 + adapters/33across/33across.go | 77 +- adapters/33across/33across_test.go | 2 +- .../exemplary/bidresponse-defaults.json | 100 + .../exemplary/instream-video-defaults.json | 108 ++ .../33acrosstest/exemplary/multi-format.json | 103 + .../exemplary/optional-params.json | 0 .../exemplary/outstream-video-defaults.json | 107 ++ .../exemplary/simple-banner.json | 14 +- .../33acrosstest/exemplary/simple-video.json | 110 ++ .../params/race/banner.json | 0 .../33acrosstest/params/race/video.json | 6 + .../supplemental/status-not-ok.json | 67 + .../supplemental/video-validation-fail.json | 32 + adapters/33across/usersync_test.go | 2 +- adapters/acuityads/acuityads.go | 190 ++ adapters/acuityads/acuityads_test.go | 11 + .../acuityadstest/exemplary/banner-app.json | 150 ++ .../acuityadstest/exemplary/banner-web.json | 144 ++ .../acuityadstest/exemplary/native-app.json | 153 ++ .../acuityadstest/exemplary/native-web.json | 139 ++ .../acuityadstest/exemplary/video-app.json | 165 ++ .../acuityadstest/exemplary/video-web.json | 156 ++ .../acuityadstest/params/race/banner.json | 4 + .../acuityadstest/params/race/native.json | 4 + .../acuityadstest/params/race/video.json | 4 + .../supplemental/empty-seatbid-array.json | 137 ++ .../supplemental/invalid-response.json | 112 ++ .../invalid-smartyads-ext-object.json | 29 + .../supplemental/status-code-bad-request.json | 93 + .../supplemental/status-code-no-content.json | 69 + .../supplemental/status-code-other-error.json | 79 + .../status-code-service-unavailable.json | 79 + adapters/acuityads/params_test.go | 51 + adapters/acuityads/usersync.go | 12 + adapters/acuityads/usersync_test.go | 34 + adapters/adapterstest/test_json.go | 6 +- adapters/adform/adform.go | 77 +- adapters/adform/adform_test.go | 122 +- .../exemplary/multiformat-impression.json | 99 + .../exemplary/single-banner-impression.json | 64 + .../exemplary/single-video-impression.json | 60 + .../adform/adformtest/params/race/video.json | 3 + adapters/adform/params_test.go | 8 + adapters/adgeneration/adgeneration.go | 13 +- adapters/adgeneration/adgeneration_test.go | 111 +- .../exemplary/single-banner.json | 8 +- .../supplemental/204-bid-response.json | 10 +- .../supplemental/400-bid-response.json | 10 +- .../supplemental/no-bid-response.json | 8 +- adapters/adhese/adhese.go | 12 +- .../adhesetest/exemplary/banner-internal.json | 7 +- adapters/adkernel/usersync_test.go | 2 +- adapters/adkernelAdn/usersync_test.go | 2 +- adapters/adman/adman.go | 140 ++ adapters/adman/adman_test.go | 12 + .../admantest/exemplary/simple-banner.json | 134 ++ .../admantest/exemplary/simple-video.json | 119 ++ .../exemplary/simple-web-banner.json | 133 ++ adapters/adman/admantest/params/banner.json | 3 + .../adman/admantest/params/race/banner.json | 3 + .../adman/admantest/params/race/video.json | 3 + adapters/adman/admantest/params/video.json | 3 + .../admantest/supplemental/bad-imp-ext.json | 42 + .../admantest/supplemental/bad_response.json | 85 + .../admantest/supplemental/no-imp-ext-1.json | 39 + .../admantest/supplemental/no-imp-ext-2.json | 39 + .../admantest/supplemental/status-204.json | 79 + .../admantest/supplemental/status-404.json | 85 + adapters/adman/params_test.go | 46 + adapters/adman/usersync.go | 13 + adapters/adman/usersync_test.go | 35 + adapters/admixer/usersync_test.go | 7 +- adapters/adoppler/adoppler.go | 36 +- adapters/adoppler/adoppler_test.go | 2 +- .../adopplertest/exemplary/custom-client.json | 80 + .../exemplary/default-client.json | 78 + .../adopplertest/exemplary/multibid.json | 60 - .../adopplertest/exemplary/multiimp.json | 207 ++ .../adopplertest/exemplary/no-bid.json | 13 - .../supplemental/bad-request.json | 71 +- .../supplemental/duplicate-imp.json | 166 +- .../supplemental/invalid-impid.json | 91 +- .../supplemental/invalid-response.json | 71 +- .../supplemental/invalid-video-ext.json | 188 +- .../supplemental/missing-adunit.json | 40 +- .../adopplertest/supplemental/no-bid.json | 50 + .../supplemental/server-error.json | 71 +- adapters/adpone/usersync.go | 2 +- adapters/adprime/adprime.go | 138 ++ adapters/adprime/adprime_test.go | 12 + .../adprimetest/exemplary/simple-banner.json | 134 ++ .../adprimetest/exemplary/simple-video.json | 119 ++ .../exemplary/simple-web-banner.json | 133 ++ .../adprime/adprimetest/params/banner.json | 3 + .../adprimetest/params/race/banner.json | 3 + .../adprimetest/params/race/video.json | 3 + .../adprime/adprimetest/params/video.json | 3 + .../adprimetest/supplemental/bad-imp-ext.json | 42 + .../supplemental/bad_response.json | 85 + .../supplemental/no-imp-ext-1.json | 39 + .../supplemental/no-imp-ext-2.json | 39 + .../adprimetest/supplemental/status-204.json | 79 + .../adprimetest/supplemental/status-404.json | 85 + adapters/adprime/params_test.go | 46 + adapters/adtarget/usersync_test.go | 5 +- adapters/adtelligent/usersync.go | 2 +- adapters/adtelligent/usersync_test.go | 2 +- adapters/aja/usersync_test.go | 5 +- adapters/amx/amx.go | 210 ++ adapters/amx/amx_test.go | 180 ++ .../amx/amxtest/exemplary/app-simple.json | 178 ++ .../amx/amxtest/exemplary/video-simple.json | 245 +++ .../amx/amxtest/exemplary/web-simple.json | 246 +++ adapters/amx/amxtest/params/race/display.json | 1 + adapters/amx/amxtest/params/race/video.json | 1 + .../amxtest/supplemental/204-response.json | 109 ++ .../amxtest/supplemental/400-response.json | 114 ++ .../amxtest/supplemental/500-response.json | 114 ++ adapters/amx/params_test.go | 47 + adapters/amx/usersync.go | 13 + adapters/amx/usersync_test.go | 23 + adapters/appnexus/appnexus.go | 62 +- adapters/appnexus/appnexus_test.go | 229 +++ .../video/simple-video.json | 132 -- .../exemplary/banner-app.json | 116 ++ .../audienceNetworktest/exemplary/banner.json | 138 -- .../exemplary/interstitial.json | 18 +- .../exemplary/native-1.1.json | 18 +- .../audienceNetworktest/exemplary/video.json | 18 +- .../supplemental/banner-format-only.json | 18 +- .../supplemental/invalid-adm.json | 103 + .../supplemental/invalid-banner-height.json | 6 +- .../supplemental/invalid-interstitial.json | 40 + .../supplemental/missing-adm-bidid.json | 107 ++ .../supplemental/missing-adm.json | 106 ++ .../supplemental/missing-banner-height.json | 6 +- .../supplemental/multi-imp.json | 30 +- .../supplemental/no-bid-204.json | 18 +- .../supplemental/no-imps.json | 22 + .../supplemental/required-buyeruid.json | 6 +- .../required-param-placementId.json | 6 +- .../required-param-publisherId.json | 6 +- .../supplemental/server-error-500.json | 87 + .../supplemental/site-not-supported.json | 38 + .../supplemental/split-placementId.json | 18 +- adapters/audienceNetwork/facebook.go | 148 +- adapters/audienceNetwork/facebook_test.go | 44 +- adapters/avocet/usersync_test.go | 2 +- .../exemplary/minimal-banner.json | 43 +- .../exemplary/simple-adm-video.json | 100 +- .../beachfronttest/exemplary/simple-mix.json | 136 +- .../exemplary/simple-nurl-video.json | 130 +- .../minimal-banner-empty_array-200.json | 2 +- .../supplemental/minimal-mobile-video.json | 112 +- .../supplemental/minimal-site-banner.json | 171 +- .../supplemental/mobile-banner.json | 28 +- .../supplemental/multi-banner.json | 24 +- .../supplemental/multi-video.json | 214 +-- .../supplemental/unmarshal-error-banner.json | 2 +- ...nmarshal-error-but-another-good-video.json | 2 +- .../supplemental/unmarshal-error-video.json | 2 +- adapters/beachfront/usersync_test.go | 2 +- adapters/beintoo/usersync_test.go | 2 +- adapters/between/between.go | 171 ++ adapters/between/between_test.go | 10 + .../betweentest/exemplary/multi-request.json | 140 ++ .../exemplary/simple-site-banner.json | 107 ++ .../betweentest/params/race/banner.json | 3 + .../supplemental/bad-bidder-ext.json | 43 + .../supplemental/bad-dsp-request-example.json | 76 + .../supplemental/bad-response-body.json | 88 + .../dsp-server-internal-error-example.json | 76 + .../supplemental/missing-host.json | 44 + .../betweentest/supplemental/missing-imp.json | 27 + .../betweentest/supplemental/no-bids.json | 69 + .../unknown-status-code-example.json | 74 + .../supplemental/zero-bid-request-error.json | 19 + adapters/bidder.go | 2 +- adapters/brightroll/brightroll.go | 81 +- adapters/brightroll/brightroll_test.go | 28 +- .../exemplary/banner-native-audio.json | 23 +- .../exemplary/banner-video-native.json | 28 +- .../exemplary/banner-video.json | 24 +- .../exemplary/simple-banner.json | 15 +- .../exemplary/simple-video.json | 15 +- .../exemplary/valid-extension.json | 15 +- .../exemplary/video-and-audio.json | 19 +- .../brightrolltest/params/race/banner.json | 2 +- .../brightrolltest/params/race/video.json | 2 +- .../supplemental/invalid-imp.json | 2 +- .../supplemental/invalid-publisher.json | 34 + adapters/colossus/colossus.go | 137 ++ adapters/colossus/colossus_test.go | 12 + .../colossustest/exemplary/simple-banner.json | 132 ++ .../colossustest/exemplary/simple-video.json | 119 ++ .../exemplary/simple-web-banner.json | 133 ++ .../colossus/colossustest/params/banner.json | 3 + .../colossustest/params/race/banner.json | 3 + .../colossustest/params/race/video.json | 3 + .../colossus/colossustest/params/video.json | 3 + .../supplemental/bad-imp-ext.json | 42 + .../supplemental/bad_response.json | 85 + .../supplemental/bad_status_code.json | 79 + .../supplemental/empty_imp_ext.json | 38 + .../supplemental/imp_ext_empty_object.json | 39 + .../supplemental/imp_ext_string.json | 39 + .../colossustest/supplemental/status-204.json | 79 + .../colossustest/supplemental/status-404.json | 85 + .../supplemental/string_imp_ext.json | 39 + adapters/colossus/params_test.go | 46 + adapters/colossus/usersync.go | 13 + adapters/colossus/usersync_test.go | 35 + adapters/connectad/connectad.go | 209 ++ adapters/connectad/connectad_test.go | 11 + .../exemplary/optional-params.json | 125 ++ .../exemplary/simple-banner.json | 122 ++ .../connectadtest/params/race/banner.json | 5 + .../connectadtest/supplemental/204.json | 86 + .../supplemental/badresponse.json | 91 + .../supplemental/banner-multi.json | 170 ++ .../connectadtest/supplemental/err500.json | 91 + .../connectadtest/supplemental/ipv6.json | 123 ++ .../connectadtest/supplemental/no_banner.json | 31 + .../connectadtest/supplemental/no_device.json | 67 + .../connectadtest/supplemental/no_dnt.json | 121 ++ .../connectadtest/supplemental/no_ext.json | 33 + .../connectadtest/supplemental/no_format.json | 35 + .../connectadtest/supplemental/wrongext.json | 38 + adapters/connectad/params_test.go | 51 + adapters/connectad/usersync.go | 12 + adapters/connectad/usersync_test.go | 35 + adapters/consumable/consumable.go | 38 +- adapters/consumable/usersync_test.go | 2 +- adapters/conversant/cnvr_legacy.go | 291 +++ adapters/conversant/cnvr_legacy_test.go | 853 +++++++++ adapters/conversant/conversant.go | 356 ++-- adapters/conversant/conversant_test.go | 849 +-------- .../conversanttest/exemplary/banner.json | 113 ++ .../conversanttest/exemplary/simple_app.json | 114 ++ .../conversanttest/exemplary/video.json | 138 ++ .../supplemental/missing_cnvrext.json | 32 + .../supplemental/missing_ext.json | 30 + .../supplemental/missing_siteid.json | 35 + .../supplemental/server_badresponse.json | 57 + .../supplemental/server_nocontent.json | 51 + .../supplemental/server_unknownstatus.json | 55 + adapters/datablocks/usersync.go | 4 +- adapters/datablocks/usersync_test.go | 2 +- adapters/dmx/dmx.go | 36 +- .../exemplary/imp-populated-banner.json | 139 ++ adapters/emx_digital/emx_digital.go | 81 +- .../exemplary/banner-and-video-app.json | 198 ++ .../exemplary/banner-and-video-site.json | 200 ++ .../emx_digitaltest/exemplary/banner-app.json | 119 ++ .../exemplary/minimal-banner.json | 11 +- .../emx_digitaltest/exemplary/video-app.json | 132 ++ .../emx_digitaltest/exemplary/video-ctv.json | 150 ++ .../emx_digitaltest/exemplary/video-site.json | 135 ++ .../emx_digitaltest/params/race/video.json | 4 + .../supplemental/add-bidfloor.json | 8 +- .../app-domain-and-url-correctly-parsed.json | 66 + .../app-storeUrl-correctly-parsed.json | 64 + .../bad-imp-banner-missing-sizes.json | 2 +- .../supplemental/bad-imp-ext-tagid-value.json | 2 +- .../bad-imp-video-missing-mimes.json | 34 + .../bad-imp-video-missing-sizes.json | 36 + .../supplemental/build-banner-object.json | 10 +- .../supplemental/build-video-object.json | 69 + .../invalid-response-no-bids.json | 5 + .../invalid-response-unmarshall-error.json | 7 +- .../supplemental/no-imps-in-request.json | 2 +- .../supplemental/server-error-code.json | 5 + .../supplemental/server-no-content.json | 79 +- .../site-domain-and-url-correctly-parsed.json | 5 + adapters/emx_digital/usersync_test.go | 2 +- adapters/engagebdr/usersync_test.go | 2 +- adapters/eplanning/eplanning.go | 25 +- .../supplemental/bad-page-site.json | 31 + .../site-page-and-url-correctly-parsed.json | 75 + adapters/eplanning/usersync.go | 2 +- adapters/eplanning/usersync_test.go | 2 +- adapters/gamma/gamma.go | 77 +- .../exemplary/banner-and-video-and-audio.json | 15 +- .../exemplary/valid-full-params.json | 2 +- .../gammatest/supplemental/bad-request.json | 2 +- .../gammatest/supplemental/missing-adm.json | 76 + .../gammatest/supplemental/missing-param.json | 2 +- .../gammatest/supplemental/missing-zone.json | 2 +- .../supplemental/nobid-signaling.json | 56 + .../supplemental/status-forbidden.json | 2 +- .../supplemental/status-no-content.json | 2 +- adapters/gamoshi/usersync_test.go | 2 +- adapters/gumgum/gumgum.go | 57 +- .../gumgum/gumgumtest/exemplary/video.json | 106 ++ .../gumgum/gumgumtest/params/race/video.json | 3 + .../supplemental/missing-video-params.json | 36 + adapters/gumgum/usersync_test.go | 2 +- adapters/improvedigital/usersync_test.go | 2 +- adapters/info.go | 17 +- adapters/info_test.go | 142 +- adapters/inmobi/inmobi.go | 127 ++ adapters/inmobi/inmobi_test.go | 10 + .../inmobitest/exemplary/simple-banner.json | 107 ++ .../inmobitest/exemplary/simple-video.json | 109 ++ .../inmobi/inmobitest/params/race/banner.json | 3 + .../inmobi/inmobitest/params/race/video.json | 3 + .../inmobi/inmobitest/supplemental/204.json | 61 + .../inmobi/inmobitest/supplemental/400.json | 67 + .../supplemental/banner-format-coersion.json | 113 ++ .../supplemental/ext-unmarshal-err.json | 28 + .../supplemental/missing-plc-error.json | 28 + .../inmobitest/supplemental/no-imp-error.json | 13 + adapters/invibes/invibes.go | 336 ++++ adapters/invibes/invibes_test.go | 11 + adapters/invibes/invibestest/amp/amp-ad.json | 76 + .../invibestest/exemplary/advanced-ad.json | 129 ++ .../invibestest/exemplary/basic-ad.json | 92 + .../invibes/invibestest/exemplary/no-ad.json | 63 + .../invibestest/exemplary/test-ad.json | 76 + .../invibestest/params/race/banner.json | 3 + .../supplemental/request-error-banner.json | 27 + .../supplemental/request-error-impext.json | 23 + .../request-error-invibesparams.json | 36 + .../supplemental/request-error-servererr.json | 67 + .../supplemental/request-error-site.json | 32 + .../request-error-statuscode.json | 63 + adapters/invibes/params_test.go | 55 + adapters/invibes/usersync.go | 17 + adapters/invibes/usersync_test.go | 32 + adapters/krushmedia/krushmedia.go | 186 ++ adapters/krushmedia/krushmedia_test.go | 11 + .../krushmediatest/exemplary/banner-app.json | 154 ++ .../krushmediatest/exemplary/banner-web.json | 148 ++ .../krushmediatest/exemplary/native-app.json | 157 ++ .../krushmediatest/exemplary/native-web.json | 144 ++ .../krushmediatest/exemplary/video-app.json | 169 ++ .../krushmediatest/exemplary/video-web.json | 161 ++ .../krushmediatest/params/race/banner.json | 3 + .../krushmediatest/params/race/native.json | 3 + .../krushmediatest/params/race/video.json | 3 + .../supplemental/invalid-ext-object.json | 29 + .../supplemental/invalid-response.json | 116 ++ .../supplemental/requires-imp-object.json | 16 + .../supplemental/status-code-bad-request.json | 91 + .../supplemental/status-code-no-content.json | 73 + .../supplemental/status-code-other-error.json | 78 + .../status-code-service-unavailable.json | 73 + adapters/krushmedia/params_test.go | 50 + adapters/krushmedia/usersync.go | 12 + adapters/krushmedia/usersync_test.go | 34 + adapters/kubient/kubient.go | 49 +- .../kubient/kubienttest/exemplary/banner.json | 8 +- .../kubient/kubienttest/exemplary/video.json | 8 +- .../supplemental/bad_response.json | 8 +- .../supplemental/missing-zoneid.json | 31 + .../kubienttest/supplemental/no-imps.json | 12 + .../kubienttest/supplemental/status_204.json | 2 + .../kubienttest/supplemental/status_400.json | 8 +- adapters/logicad/logicad.go | 155 ++ adapters/logicad/logicad_test.go | 10 + .../logicad/logicadtest/exemplary/banner.json | 92 + .../logicadtest/params/race/banner.json | 3 + .../logicadtest/supplemental/checkImp.json | 15 + .../logicad/logicadtest/supplemental/ext.json | 31 + .../logicadtest/supplemental/missingtid.json | 33 + .../supplemental/multiImpSameTid.json | 112 ++ .../supplemental/responseCode.json | 72 + .../supplemental/responseNoBid.json | 66 + .../logicadtest/supplemental/responsebid.json | 73 + .../logicadtest/supplemental/site.json | 98 + adapters/logicad/params_test.go | 45 + adapters/logicad/usersync.go | 12 + adapters/logicad/usersync_test.go | 31 + adapters/marsmedia/usersync_test.go | 2 +- adapters/nanointeractive/usersync_test.go | 24 +- adapters/nobid/nobid.go | 122 ++ adapters/nobid/nobid_test.go | 12 + .../nobid/nobidtest/exemplary/banner.json | 84 + .../nobidtest/supplemental/bad-mediatype.json | 47 + .../nobidtest/supplemental/bad-request.json | 51 + .../nobidtest/supplemental/bad-response.json | 71 + .../nobidtest/supplemental/missing-imps.json | 14 + .../nobidtest/supplemental/no-content.json | 46 + .../supplemental/notok-response.json | 51 + adapters/nobid/params_test.go | 53 + adapters/nobid/usersync.go | 12 + adapters/nobid/usersync_test.go | 34 + adapters/openx/openx.go | 10 +- .../openxtest/exemplary/optional-params.json | 4 +- adapters/pubmatic/pubmatic.go | 17 +- adapters/pubmatic/usersync_test.go | 3 +- adapters/pubnative/pubnative.go | 1 - .../exemplary/simple-banner.json | 1 + adapters/rhythmone/usersync_test.go | 2 +- adapters/rubicon/rubicon.go | 100 +- adapters/rubicon/rubicon_test.go | 14 +- adapters/sharethrough/butler.go | 15 +- adapters/silvermob/params_test.go | 56 + adapters/silvermob/silvermob.go | 188 ++ adapters/silvermob/silvermob_test.go | 11 + .../silvermobtest/exemplary/banner-app.json | 163 ++ .../exemplary/banner-multi-app.json | 293 +++ .../silvermobtest/exemplary/native-app.json | 159 ++ .../silvermobtest/exemplary/video-app.json | 171 ++ .../silvermobtest/params/race/banner.json | 4 + .../silvermobtest/params/race/native.json | 4 + .../silvermobtest/params/race/video.json | 4 + .../supplemental/empty-seatbid-array.json | 139 ++ .../supplemental/invalid-response.json | 118 ++ .../invalid-silvermob-ext-object.json | 28 + .../supplemental/status-code-bad-request.json | 99 + .../supplemental/status-code-no-content.json | 83 + .../supplemental/status-code-other-error.json | 87 + .../status-code-service-unavailable.json | 87 + adapters/smaato/image.go | 53 + adapters/smaato/image_test.go | 44 + adapters/smaato/params_test.go | 65 + adapters/smaato/richmedia.go | 52 + adapters/smaato/richmedia_test.go | 39 + adapters/smaato/smaato.go | 311 +++ adapters/smaato/smaato_test.go | 11 + .../exemplary/simple-banner-richMedia.json | 194 ++ .../smaatotest/exemplary/simple-banner.json | 190 ++ .../smaato/smaatotest/exemplary/video.json | 187 ++ adapters/smaato/smaatotest/params/banner.json | 4 + .../supplemental/bad-adm-response.json | 172 ++ .../smaatotest/supplemental/bad-ext-req.json | 54 + .../bad-imp-banner-format-req.json | 61 + .../supplemental/bad-user-ext-data-req.json | 67 + .../supplemental/bad-user-ext-req.json | 57 + .../supplemental/no-consent-info.json | 137 ++ .../smaatotest/supplemental/no-imp-req.json | 17 + adapters/smartadserver/params_test.go | 61 + adapters/smartadserver/smartadserver.go | 179 ++ adapters/smartadserver/smartadserver_test.go | 11 + .../exemplary/multi-banner.json | 175 ++ .../exemplary/simple-banner.json | 94 + .../exemplary/simple-video.json | 100 + .../smartadservertest/params/race/banner.json | 7 + .../smartadservertest/params/race/video.json | 7 + .../request-no-bidder-object.json | 21 + .../supplemental/request-no-ext-object.json | 19 + .../supplemental/request-no-imp.json | 13 + .../supplemental/request-site-recreated.json | 99 + .../response-200-without-body.json | 62 + .../supplemental/response-204.json | 56 + .../supplemental/response-400.json | 62 + .../supplemental/response-500.json | 62 + adapters/smartadserver/usersync.go | 12 + adapters/smartadserver/usersync_test.go | 35 + adapters/smartyads/params_test.go | 52 + adapters/smartyads/smartyads.go | 202 ++ adapters/smartyads/smartyads_test.go | 11 + .../smartyadstest/exemplary/banner-app.json | 157 ++ .../smartyadstest/exemplary/banner-web.json | 151 ++ .../smartyadstest/exemplary/native-app.json | 160 ++ .../smartyadstest/exemplary/native-web.json | 146 ++ .../smartyadstest/exemplary/video-app.json | 172 ++ .../smartyadstest/exemplary/video-web.json | 163 ++ .../smartyadstest/params/race/banner.json | 5 + .../smartyadstest/params/race/native.json | 5 + .../smartyadstest/params/race/video.json | 5 + .../supplemental/empty-seatbid-array.json | 144 ++ .../supplemental/invalid-response.json | 119 ++ .../invalid-smartyads-ext-object.json | 29 + .../supplemental/status-code-bad-request.json | 94 + .../supplemental/status-code-no-content.json | 80 + .../supplemental/status-code-other-error.json | 80 + .../status-code-service-unavailable.json | 80 + adapters/smartyads/usersync.go | 12 + adapters/smartyads/usersync_test.go | 34 + adapters/syncer.go | 2 +- adapters/syncer_test.go | 2 +- adapters/telaria/telaria.go | 20 +- .../telariatest/exemplary/video-app.json | 226 ++- .../telariatest/exemplary/video-web.json | 223 ++- .../ucfunneltest/exemplary/ucfunnel.json | 5 +- adapters/unruly/usersync_test.go | 2 +- adapters/valueimpression/usersync_test.go | 2 +- adapters/visx/usersync_test.go | 2 +- adapters/yieldmo/yieldmo.go | 12 +- .../yieldmotest/exemplary/app-banner.json | 97 + .../yieldmotest/exemplary/app_video.json | 95 + .../yieldmotest/exemplary/simple_video.json | 95 + adapters/zeroclickfraud/usersync_test.go | 2 +- analytics/clients/http.go | 12 + analytics/config/config.go | 23 + analytics/config/config_test.go | 48 + analytics/core.go | 9 + analytics/event.go | 38 + analytics/filesystem/file_module.go | 40 +- analytics/filesystem/file_module_test.go | 16 + analytics/pubstack/README.md | 28 + analytics/pubstack/config.go | 51 + analytics/pubstack/config_test.go | 102 + .../pubstack/eventchannel/eventchannel.go | 137 ++ .../eventchannel/eventchannel_test.go | 181 ++ analytics/pubstack/eventchannel/sender.go | 45 + .../pubstack/eventchannel/sender_test.go | 40 + analytics/pubstack/helpers/json.go | 88 + analytics/pubstack/helpers/json_test.go | 61 + .../pubstack/mocks/mock_openrtb_request.json | 64 + .../pubstack/mocks/mock_openrtb_response.json | 91 + analytics/pubstack/pubstack_module.go | 276 +++ analytics/pubstack/pubstack_module_test.go | 225 +++ config/accounts.go | 79 + config/accounts_test.go | 243 +++ config/config.go | 316 ++- config/config_test.go | 253 ++- config/requestvalidation.go | 55 + config/requestvalidation_test.go | 145 ++ config/stored_requests.go | 320 ++-- config/stored_requests_test.go | 231 ++- config/util/loggers.go | 6 +- currencies/converter_info.go | 15 +- currencies/rate_converter.go | 165 +- currencies/rate_converter_test.go | 633 +++--- currencies/rates_test.go | 1 + devcontainer.md | 67 + docs/bidders/adtarget.md | 5 - docs/bidders/appnexus.md | 45 - docs/bidders/audienceNetwork.md | 8 - docs/bidders/avocet.md | 5 - docs/bidders/beachfront.md | 13 - docs/bidders/emx_digital.md | 10 - docs/bidders/kidoz.md | 9 - docs/bidders/openx.md | 62 - docs/bidders/pubmatic.md | 33 - docs/bidders/pubnative.md | 62 - docs/bidders/rubicon.md | 7 - docs/bidders/smartrtb.md | 39 - docs/bidders/sovrn.md | 3 - docs/bidders/tappx.md | 13 - ...Server Event Notifications - Tech Spec.pdf | Bin 89983 -> 0 bytes docs/developers/add-new-analytics-module.md | 33 - docs/developers/add-new-bidder.md | 117 -- docs/developers/cookie-syncs.md | 30 - docs/developers/currency-converter.md | 56 - docs/developers/default-request.md | 44 - docs/developers/features.md | 12 + docs/developers/gdpr.md | 31 - docs/developers/stored-requests.md | 4 +- docs/endpoints.md | 1 + docs/endpoints/bidders/params.md | 24 - docs/endpoints/cookieSync.md | 55 - docs/endpoints/currency_rates.md | 111 -- docs/endpoints/info/bidders.md | 23 - docs/endpoints/info/bidders/bidderName.md | 43 - docs/endpoints/openrtb2/amp.md | 127 -- docs/endpoints/openrtb2/auction.md | 786 -------- docs/endpoints/setuid.md | 26 - docs/endpoints/status.md | 9 - endpoints/auction.go | 11 +- endpoints/auction_test.go | 10 +- endpoints/cookie_sync.go | 50 +- endpoints/cookie_sync_test.go | 4 +- endpoints/currency_rates.go | 7 +- endpoints/currency_rates_test.go | 51 +- endpoints/events/account_test.go | 161 ++ endpoints/events/event.go | 338 ++++ endpoints/events/event_test.go | 664 +++++++ endpoints/events/vtrack.go | 300 +++ endpoints/events/vtrack_test.go | 692 +++++++ endpoints/openrtb2/amp_auction.go | 124 +- endpoints/openrtb2/amp_auction_test.go | 260 ++- endpoints/openrtb2/auction.go | 257 ++- endpoints/openrtb2/auction_benchmark_test.go | 4 +- endpoints/openrtb2/auction_test.go | 1693 +++++++++++++---- endpoints/openrtb2/ctv_auction.go | 45 +- .../no-account/not-required-no-acct.json | 84 + .../no-account/required-no-acct.json | 68 + .../account-required/no-acct.json | 66 - .../{with-acct.json => valid-acct.json} | 12 +- .../required-blacklisted-acct.json | 91 + .../with-account/required-with-acct.json | 86 + .../aliased/multiple-alias.json | 93 + .../sample-requests/aliased/simple.json | 28 +- .../blacklisted/blacklisted-acct.json | 88 - .../blacklisted-app-publisher.json | 90 + .../blacklisted/blacklisted-app.json | 11 +- .../blacklisted-site-publisher.json | 90 + .../disabled/bad/bad-alias.json | 13 +- .../disabled/bad/bad-bidder.json | 13 +- .../disabled/good/partial.json | 26 +- ...valid-context-allowed-with-ext-bidder.json | 50 + ...id-context-allowed-with-prebid-bidder.json | 54 + .../asset-data-invalid-type.json | 30 +- .../invalid-native/asset-data-no-type.json | 28 +- .../invalid-native/asset-empty.json | 28 +- .../invalid-native/asset-img-h-negative.json | 31 +- .../asset-img-hmin-negative.json | 31 +- .../invalid-native/asset-img-w-negative.json | 31 +- .../asset-img-wmin-negative.json | 31 +- .../invalid-native/asset-mixed-type.json | 34 +- .../invalid-native/asset-title-empty.json | 25 + .../invalid-native/asset-title-no-length.json | 9 - .../asset-video-mimes-empty.json | 33 +- .../asset-video-no-maxduration.json | 32 +- .../invalid-native/asset-video-no-mimes.json | 32 +- .../asset-video-no-minduration.json | 32 +- .../asset-video-no-protocols.json | 32 +- .../asset-video-protocols-empty.json | 33 +- .../asset-video-protocols-invalid.json | 33 +- .../invalid-native/assets-with-dup-ids.json | 33 +- .../assets-with-partial-ids.json | 37 +- .../contextsubtype-invalid.json | 46 +- .../contextsubtype-negative.json | 46 +- .../invalid-native/empty-object.json | 26 +- .../sample-requests/invalid-native/empty.json | 25 + .../invalid-native/eventtracker-empty.json | 48 +- .../eventtracker-event-large.json | 25 + .../eventtracker-methods-empty.json | 51 +- .../eventtracker-methods-large.json | 51 +- .../eventtracker-type-large.json | 34 - .../request-context-invalid.json | 31 +- .../request-plcmttype-invalid.json | 31 +- .../invalid-stored/bad_incoming_1.json | 46 +- .../invalid-stored/bad_incoming_2.json | 41 - .../invalid-stored/bad_incoming_imp.json | 7 +- .../invalid-stored/bad_stored_imp.json | 7 +- .../invalid-stored/bad_stored_req.json | 35 +- .../invalid-whole/alias-bidder-self.json | 14 +- .../invalid-whole/alias-unknown-core.json | 10 +- .../invalid-whole/app-bad-ext.json | 41 - .../sample-requests/invalid-whole/array.json | 4 - .../invalid-whole/audio-mimes-empty.json | 8 +- .../invalid-whole/banner-h-only.json | 8 +- .../invalid-whole/banner-h-zero.json | 23 - .../invalid-whole/banner-hmax.json | 8 +- .../invalid-whole/banner-hmin.json | 28 +- .../invalid-whole/banner-null.json | 15 - .../invalid-whole/banner-w-only.json | 8 +- .../invalid-whole/banner-w-zero.json | 23 - .../invalid-whole/banner-wmax.json | 8 +- .../invalid-whole/banner-wmin.json | 8 +- .../bid-adjustment-invalid-bidder.json | 8 +- .../bid-adjustment-negative.json | 8 +- .../invalid-whole/boolean.json | 4 - .../invalid-whole/cache-nothing.json | 10 +- .../invalid-whole/deal-no-id.json | 10 +- .../invalid-whole/digitrust.json | 8 +- .../invalid-whole/empty-object.json | 6 +- .../sample-requests/invalid-whole/float.json | 4 - .../invalid-whole/format-empty-array.json | 8 +- .../invalid-whole/format-empty-object.json | 8 +- .../invalid-whole/format-no-height.json | 8 +- .../invalid-whole/format-no-hratio.json | 38 +- .../invalid-whole/format-two-widths.json | 8 +- .../invalid-whole/imp-empty-array.json | 8 +- .../invalid-whole/imp-empty-object.json | 8 +- .../invalid-whole/imp-ext-empty.json | 10 +- .../invalid-whole/imp-ext-invalid-params.json | 10 +- .../invalid-whole/imp-ext-unknown-bidder.json | 10 +- .../invalid-whole/imp-id-duplicates.json | 8 +- .../invalid-whole/imp-no-ext.json | 10 +- .../invalid-whole/imp-no-type.json | 8 +- .../invalid-whole/integer.json | 4 - .../invalid-whole/interstital-bad-perc.json | 88 +- .../invalid-whole/interstitial-empty.json | 74 +- .../invalid-source.json} | 41 +- .../invalid-whole/malformed-bid-request.json | 6 + .../invalid-whole/metric-empty-object.json | 26 +- .../invalid-whole/native-empty.json | 8 +- .../invalid-whole/no-site-or-app.json | 10 +- .../sample-requests/invalid-whole/null.json | 6 +- .../invalid-whole/only-request-id.json | 8 +- .../invalid-whole/regs-ext-gdpr-invalid.json | 14 +- .../invalid-whole/regs-ext-gdpr-string.json | 8 +- .../invalid-whole/regs-ext-malformed.json | 79 +- .../invalid-whole/site-app-both.json | 10 +- .../invalid-whole/site-empty.json | 10 +- .../invalid-whole/site-ext-amp.json | 8 +- .../invalid-whole/storedrequest-id-int.json | 10 +- .../invalid-whole/tmax-negative.json | 8 +- .../invalid-whole/unknown-bidder.json | 66 +- .../invalid-whole/user-ext-consent-int.json | 24 +- .../user-ext-eids-eids-uids-empty.json | 8 +- .../invalid-whole/user-ext-eids-empty.json | 8 +- .../user-ext-eids-id-uids-empty.json | 8 +- .../user-ext-eids-source-empty.json | 8 +- .../user-ext-eids-source-unique.json | 8 +- .../user-ext-eids-uids-id-empty.json | 8 +- .../user-ext-prebid-buyeruids-empty.json | 8 +- .../user-ext-prebid-buyeruids-unknown.json | 8 +- .../invalid-whole/user-ext-prebid-empty.json | 8 +- .../invalid-whole/user-gdpr-badtype.json | 46 - ...id.json => user-gdpr-consent-invalid.json} | 15 +- .../invalid-whole/video-empty.json | 8 +- .../invalid-whole/video-mimes-empty.json | 8 +- .../valid-native/asset-img-no-hmin.json | 44 +- .../valid-native/asset-img-no-wmin.json | 44 +- .../valid-native/asset-with-id.json | 43 +- .../valid-native/asset-with-no-id.json | 44 +- .../valid-native/assets-with-unique-ids.json | 49 +- .../valid-native/request-no-context.json | 46 +- .../valid-native/request-plcmttype-empty.json | 46 +- .../valid-native/sample-v1.1.json | 31 - .../valid-native/sample-v1.2.json | 34 - .../video-asset-event-tracker.json | 41 + .../valid-native/with-video-asset.json | 41 + .../valid-whole/exemplary/all-ext.json | 43 +- .../valid-whole/exemplary/prebid-test-ad.json | 23 +- .../supplementary/aliased-buyeruids.json | 81 +- .../valid-whole/supplementary/aliases.json | 53 +- .../valid-whole/supplementary/app.json | 36 - .../supplementary/bid-adjustments.json | 32 - .../valid-whole/supplementary/cache-bids.json | 29 - .../valid-whole/supplementary/cache-vast.json | 28 - .../supplementary/ccpa-invalid.json | 41 - .../valid-whole/supplementary/digitrust.json | 77 +- .../supplementary/gdpr-no-consentstring.json | 83 +- .../valid-whole/supplementary/gdpr.json | 87 +- .../interstitial-device-only.json | 40 - .../interstitial-no-extension.json | 35 - .../supplementary/interstitial.json | 48 - .../valid-whole/supplementary/site-amp.json | 83 +- .../supplementary/site-has-dnt.json | 53 + .../supplementary/site-has-ipv4.json | 47 + .../supplementary/site-has-ipv6.json | 47 + .../valid-whole/supplementary/site.json | 77 +- .../valid-whole/supplementary/timeout.json | 30 - .../valid-whole/supplementary/user.json | 65 +- .../video_valid_sample_appendbiddernames.json | 86 + endpoints/openrtb2/video_auction.go | 116 +- endpoints/openrtb2/video_auction_test.go | 218 ++- endpoints/setuid_test.go | 4 +- exchange/adapter_map.go | 40 +- exchange/auction.go | 77 +- exchange/auction_test.go | 236 ++- exchange/bidder.go | 102 +- exchange/bidder_test.go | 292 ++- exchange/bidder_validate_bids.go | 8 +- exchange/bidder_validate_bids_test.go | 10 +- exchange/cachetest/customcachekey.json | 12 +- .../cachetest/customcachekey_no_bidders.json | 6 +- .../cachetest/customcachekey_no_winners.json | 12 +- exchange/cachetest/debuglog_disabled.json | 12 +- exchange/cachetest/debuglog_enabled.json | 18 +- .../debuglog_enabled_no_winners_nor_bids.json | 54 + exchange/cachetest/defaultbanner.json | 12 +- .../cachetest/defaultbanner_no_bidders.json | 6 +- .../cachetest/defaultbanner_no_winners.json | 12 +- exchange/cachetest/defaultvideo.json | 12 +- .../cachetest/defaultvideo_no_bidders.json | 6 +- .../cachetest/defaultvideo_no_winners.json | 12 +- exchange/cachetest/multibid.json | 30 +- exchange/cachetest/multibid_no_bidders.json | 12 +- exchange/cachetest/multibid_no_winners.json | 30 +- .../customcachekeytest/customcachekey.json | 14 +- .../customcachekey_no_bidders.json | 8 +- .../customcachekey_no_winners.json | 14 +- exchange/exchange.go | 394 ++-- exchange/exchange_test.go | 1216 ++++++++++-- .../exchangetest/append-bidder-names.json | 222 +++ .../exchangetest/ccpa-nosale-any-bidder.json | 75 + .../ccpa-nosale-specific-bidder.json | 75 + exchange/exchangetest/debuglog_disabled.json | 4 +- exchange/exchangetest/debuglog_enabled.json | 4 +- .../debuglog_enabled_no_bids.json | 72 + ...rstpartydata-imp-ext-multiple-bidders.json | 173 ++ ...ydata-imp-ext-multiple-prebid-bidders.json | 179 ++ .../firstpartydata-imp-ext-one-bidder.json | 103 + ...stpartydata-imp-ext-one-prebid-bidder.json | 108 ++ .../exchangetest/gdpr-geo-eu-off-device.json | 65 + exchange/exchangetest/gdpr-geo-eu-off.json | 61 + .../gdpr-geo-eu-on-featureflag-off.json | 62 + exchange/exchangetest/gdpr-geo-eu-on.json | 61 + exchange/exchangetest/gdpr-geo-usa-off.json | 61 + exchange/exchangetest/gdpr-geo-usa-on.json | 61 + .../request-multi-bidders-debug-info.json | 3 - .../exchangetest/targeting-cache-vast.json | 2 +- .../exchangetest/targeting-cache-zero.json | 2 +- .../impcustomcachekeytest/multiImpVast.json | 34 +- .../impcustomcachekeytest/multiImpVideo.json | 101 + .../multiImpVideoNoIncludeBidderKeys.json | 86 + exchange/legacy.go | 4 +- exchange/legacy_test.go | 22 +- exchange/price_granularity.go | 25 +- exchange/price_granularity_test.go | 125 +- exchange/targeting.go | 9 +- exchange/targeting_test.go | 240 ++- exchange/utils.go | 330 +++- exchange/utils_test.go | 1123 ++++++++++- gdpr/gdpr.go | 11 +- gdpr/impl.go | 54 +- gdpr/impl_test.go | 224 ++- gdpr/vendorlist-fetching.go | 115 +- gdpr/vendorlist-fetching_test.go | 893 +++++++-- go.mod | 3 +- go.sum | 7 +- macros/macros.go | 1 + main.go | 14 +- main_test.go | 14 +- openrtb_ext/bid.go | 4 + openrtb_ext/bid_request_video.go | 7 + openrtb_ext/bid_response_video.go | 6 +- openrtb_ext/bidders.go | 35 + openrtb_ext/bidders_test.go | 61 + openrtb_ext/deal_tier.go | 61 + openrtb_ext/deal_tier_test.go | 98 + openrtb_ext/imp.go | 26 +- openrtb_ext/imp_acuityads.go | 6 + openrtb_ext/imp_adform.go | 11 +- openrtb_ext/imp_adman.go | 6 + openrtb_ext/imp_adoppler.go | 1 + openrtb_ext/imp_adprime.go | 6 + openrtb_ext/imp_amx.go | 7 + openrtb_ext/imp_between.go | 5 + openrtb_ext/imp_colossus.go | 6 + openrtb_ext/imp_connectad.go | 7 + openrtb_ext/imp_conversant.go | 13 + openrtb_ext/imp_inmobi.go | 5 + openrtb_ext/imp_invibes.go | 12 + openrtb_ext/imp_krushmedia.go | 6 + openrtb_ext/imp_kubient.go | 6 + openrtb_ext/imp_logicad.go | 5 + openrtb_ext/imp_nobid.go | 6 + openrtb_ext/imp_openx.go | 1 + openrtb_ext/imp_silvermob.go | 7 + openrtb_ext/imp_smaato.go | 9 + openrtb_ext/imp_smartadserver.go | 9 + openrtb_ext/imp_smartyads.go | 8 + openrtb_ext/imp_telaria.go | 7 +- openrtb_ext/request.go | 76 +- openrtb_ext/request_test.go | 52 +- openrtb_ext/user.go | 4 +- pbs/pbsrequest.go | 11 +- pbsmetrics/config/metrics.go | 69 +- pbsmetrics/config/metrics_test.go | 14 + pbsmetrics/go_metrics.go | 157 +- pbsmetrics/go_metrics_test.go | 347 ++++ pbsmetrics/metrics.go | 100 + pbsmetrics/metrics_mock.go | 30 + pbsmetrics/prometheus/preload.go | 99 +- pbsmetrics/prometheus/prometheus.go | 293 ++- pbsmetrics/prometheus/prometheus_test.go | 486 ++++- pbsmetrics/prometheus/type_conversion.go | 36 + prebid/prebid.go | 82 - prebid_cache_client/client.go | 61 +- prebid_cache_client/client_test.go | 79 +- privacy/ccpa/consentwriter.go | 25 + privacy/ccpa/consentwriter_test.go | 51 + privacy/ccpa/parsedpolicy.go | 137 ++ privacy/ccpa/parsedpolicy_test.go | 391 ++++ privacy/ccpa/policy.go | 213 ++- privacy/ccpa/policy_test.go | 630 +++--- privacy/enforcement.go | 31 +- privacy/enforcement_test.go | 141 +- privacy/enforcer.go | 43 + privacy/enforcer_test.go | 18 + privacy/gdpr/consentwriter.go | 44 + privacy/gdpr/consentwriter_test.go | 101 + privacy/gdpr/policy.go | 40 +- privacy/gdpr/policy_test.go | 113 +- privacy/lmt/policy.go | 14 +- privacy/lmt/policy_test.go | 68 +- privacy/policies.go | 52 +- privacy/policies_test.go | 119 -- privacy/scrubber.go | 52 +- privacy/scrubber_test.go | 218 +-- privacy/writer.go | 18 + privacy/writer_test.go | 25 + router/admin.go | 5 +- router/router.go | 36 +- static/bidder-info/33across.yaml | 2 + static/bidder-info/acuityads.yaml | 14 + static/bidder-info/adform.yaml | 2 + static/bidder-info/adman.yaml | 11 + static/bidder-info/adprime.yaml | 11 + static/bidder-info/adtelligent.yaml | 1 + static/bidder-info/amx.yaml | 11 + static/bidder-info/audienceNetwork.yaml | 5 - static/bidder-info/between.yaml | 9 + static/bidder-info/colossus.yaml | 11 + static/bidder-info/connectad.yaml | 9 + static/bidder-info/emx_digital.yaml | 5 + static/bidder-info/grid.yaml | 6 +- static/bidder-info/gumgum.yaml | 1 + static/bidder-info/inmobi.yaml | 8 + static/bidder-info/invibes.yaml | 6 + static/bidder-info/krushmedia.yaml | 13 + static/bidder-info/logicad.yaml | 10 + static/bidder-info/nobid.yaml | 11 + static/bidder-info/silvermob.yaml | 8 + static/bidder-info/smaato.yaml | 11 + static/bidder-info/smartadserver.yaml | 11 + static/bidder-info/smartyads.yaml | 14 + static/bidder-info/yieldmo.yaml | 5 + static/bidder-params/acuityads.json | 19 + static/bidder-params/adform.json | 14 + static/bidder-params/adman.json | 15 + static/bidder-params/adoppler.json | 4 + static/bidder-params/adprime.json | 14 + static/bidder-params/amx.json | 16 + static/bidder-params/between.json | 14 + static/bidder-params/colossus.json | 14 + static/bidder-params/connectad.json | 24 + static/bidder-params/conversant.json | 4 - static/bidder-params/inmobi.json | 13 + static/bidder-params/invibes.json | 30 + static/bidder-params/krushmedia.json | 13 + static/bidder-params/kubient.json | 8 +- static/bidder-params/logicad.json | 13 + static/bidder-params/nobid.json | 18 + static/bidder-params/openx.json | 22 +- static/bidder-params/silvermob.json | 17 + static/bidder-params/smaato.json | 17 + static/bidder-params/smartadserver.json | 35 + static/bidder-params/smartyads.json | 21 + .../category-mapping/freewheel/freewheel.json | 78 +- static/tcf1/fallback_gvl.json | 1 + .../backends/db_fetcher/fetcher.go | 4 + .../backends/empty_fetcher/fetcher.go | 5 + .../backends/file_fetcher/fetcher.go | 15 + .../backends/file_fetcher/fetcher_test.go | 14 + .../file_fetcher/test/accounts/valid.json | 4 + .../backends/http_fetcher/fetcher.go | 92 +- .../backends/http_fetcher/fetcher_test.go | 173 +- stored_requests/caches/cachestest/reliable.go | 65 +- stored_requests/caches/memory/cache.go | 61 +- stored_requests/caches/memory/cache_test.go | 44 +- stored_requests/caches/nil_cache/nil_cache.go | 9 +- stored_requests/config/config.go | 146 +- stored_requests/config/config_test.go | 212 +-- stored_requests/data/by_id/accounts/test.json | 14 + stored_requests/events/api/api_test.go | 32 +- stored_requests/events/events.go | 10 +- stored_requests/events/events_test.go | 32 +- stored_requests/events/http/http.go | 17 +- stored_requests/events/http/http_test.go | 305 ++- stored_requests/events/postgres/database.go | 225 +++ .../events/postgres/database_test.go | 444 +++++ stored_requests/events/postgres/polling.go | 160 -- .../events/postgres/polling_test.go | 65 - stored_requests/events/postgres/startup.go | 61 - .../events/postgres/startup_test.go | 119 -- stored_requests/fetcher.go | 78 +- stored_requests/fetcher_test.go | 158 +- stored_requests/multifetcher.go | 15 + stored_requests/multifetcher_test.go | 51 + usersync/usersyncers/syncer.go | 22 + usersync/usersyncers/syncer_test.go | 16 + util/httputil/httputil.go | 99 + util/httputil/httputil_test.go | 327 ++++ util/iputil/parse.go | 27 + util/iputil/parse_test.go | 30 + util/iputil/validator.go | 48 + util/iputil/validator_test.go | 222 +++ util/maputil/maputil.go | 21 + util/maputil/maputil_test.go | 113 ++ util/task/ticker_task.go | 53 + util/task/ticker_task_test.go | 63 + util/timeutil/time.go | 16 + 960 files changed, 53544 insertions(+), 11398 deletions(-) create mode 100644 .devcontainer/Dockerfile create mode 100644 .devcontainer/devcontainer.json create mode 100644 .github/workflows/validate.yml create mode 100644 account/account.go create mode 100644 account/account_test.go create mode 100644 adapters/33across/33acrosstest/exemplary/bidresponse-defaults.json create mode 100644 adapters/33across/33acrosstest/exemplary/instream-video-defaults.json create mode 100644 adapters/33across/33acrosstest/exemplary/multi-format.json rename adapters/33across/{33across => 33acrosstest}/exemplary/optional-params.json (100%) create mode 100644 adapters/33across/33acrosstest/exemplary/outstream-video-defaults.json rename adapters/33across/{33across => 33acrosstest}/exemplary/simple-banner.json (85%) create mode 100644 adapters/33across/33acrosstest/exemplary/simple-video.json rename adapters/33across/{33across => 33acrosstest}/params/race/banner.json (100%) create mode 100644 adapters/33across/33acrosstest/params/race/video.json create mode 100644 adapters/33across/33acrosstest/supplemental/status-not-ok.json create mode 100644 adapters/33across/33acrosstest/supplemental/video-validation-fail.json create mode 100644 adapters/acuityads/acuityads.go create mode 100644 adapters/acuityads/acuityads_test.go create mode 100644 adapters/acuityads/acuityadstest/exemplary/banner-app.json create mode 100644 adapters/acuityads/acuityadstest/exemplary/banner-web.json create mode 100644 adapters/acuityads/acuityadstest/exemplary/native-app.json create mode 100644 adapters/acuityads/acuityadstest/exemplary/native-web.json create mode 100644 adapters/acuityads/acuityadstest/exemplary/video-app.json create mode 100644 adapters/acuityads/acuityadstest/exemplary/video-web.json create mode 100644 adapters/acuityads/acuityadstest/params/race/banner.json create mode 100644 adapters/acuityads/acuityadstest/params/race/native.json create mode 100644 adapters/acuityads/acuityadstest/params/race/video.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/empty-seatbid-array.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/invalid-response.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/invalid-smartyads-ext-object.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/status-code-bad-request.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/status-code-no-content.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/status-code-other-error.json create mode 100644 adapters/acuityads/acuityadstest/supplemental/status-code-service-unavailable.json create mode 100644 adapters/acuityads/params_test.go create mode 100644 adapters/acuityads/usersync.go create mode 100644 adapters/acuityads/usersync_test.go create mode 100644 adapters/adform/adformtest/exemplary/multiformat-impression.json create mode 100644 adapters/adform/adformtest/exemplary/single-banner-impression.json create mode 100644 adapters/adform/adformtest/exemplary/single-video-impression.json create mode 100644 adapters/adform/adformtest/params/race/video.json create mode 100644 adapters/adman/adman.go create mode 100644 adapters/adman/adman_test.go create mode 100644 adapters/adman/admantest/exemplary/simple-banner.json create mode 100644 adapters/adman/admantest/exemplary/simple-video.json create mode 100644 adapters/adman/admantest/exemplary/simple-web-banner.json create mode 100644 adapters/adman/admantest/params/banner.json create mode 100644 adapters/adman/admantest/params/race/banner.json create mode 100644 adapters/adman/admantest/params/race/video.json create mode 100644 adapters/adman/admantest/params/video.json create mode 100644 adapters/adman/admantest/supplemental/bad-imp-ext.json create mode 100644 adapters/adman/admantest/supplemental/bad_response.json create mode 100644 adapters/adman/admantest/supplemental/no-imp-ext-1.json create mode 100644 adapters/adman/admantest/supplemental/no-imp-ext-2.json create mode 100644 adapters/adman/admantest/supplemental/status-204.json create mode 100644 adapters/adman/admantest/supplemental/status-404.json create mode 100644 adapters/adman/params_test.go create mode 100644 adapters/adman/usersync.go create mode 100644 adapters/adman/usersync_test.go create mode 100644 adapters/adoppler/adopplertest/exemplary/custom-client.json create mode 100644 adapters/adoppler/adopplertest/exemplary/default-client.json delete mode 100644 adapters/adoppler/adopplertest/exemplary/multibid.json create mode 100644 adapters/adoppler/adopplertest/exemplary/multiimp.json delete mode 100644 adapters/adoppler/adopplertest/exemplary/no-bid.json create mode 100644 adapters/adoppler/adopplertest/supplemental/no-bid.json create mode 100644 adapters/adprime/adprime.go create mode 100644 adapters/adprime/adprime_test.go create mode 100644 adapters/adprime/adprimetest/exemplary/simple-banner.json create mode 100644 adapters/adprime/adprimetest/exemplary/simple-video.json create mode 100644 adapters/adprime/adprimetest/exemplary/simple-web-banner.json create mode 100644 adapters/adprime/adprimetest/params/banner.json create mode 100644 adapters/adprime/adprimetest/params/race/banner.json create mode 100644 adapters/adprime/adprimetest/params/race/video.json create mode 100644 adapters/adprime/adprimetest/params/video.json create mode 100644 adapters/adprime/adprimetest/supplemental/bad-imp-ext.json create mode 100644 adapters/adprime/adprimetest/supplemental/bad_response.json create mode 100644 adapters/adprime/adprimetest/supplemental/no-imp-ext-1.json create mode 100644 adapters/adprime/adprimetest/supplemental/no-imp-ext-2.json create mode 100644 adapters/adprime/adprimetest/supplemental/status-204.json create mode 100644 adapters/adprime/adprimetest/supplemental/status-404.json create mode 100644 adapters/adprime/params_test.go create mode 100644 adapters/amx/amx.go create mode 100644 adapters/amx/amx_test.go create mode 100644 adapters/amx/amxtest/exemplary/app-simple.json create mode 100644 adapters/amx/amxtest/exemplary/video-simple.json create mode 100644 adapters/amx/amxtest/exemplary/web-simple.json create mode 100644 adapters/amx/amxtest/params/race/display.json create mode 100644 adapters/amx/amxtest/params/race/video.json create mode 100644 adapters/amx/amxtest/supplemental/204-response.json create mode 100644 adapters/amx/amxtest/supplemental/400-response.json create mode 100644 adapters/amx/amxtest/supplemental/500-response.json create mode 100644 adapters/amx/params_test.go create mode 100644 adapters/amx/usersync.go create mode 100644 adapters/amx/usersync_test.go delete mode 100644 adapters/appnexus/appnexusplatformtest/video/simple-video.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/exemplary/banner-app.json delete mode 100644 adapters/audienceNetwork/audienceNetworktest/exemplary/banner.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-adm.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-interstitial.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm-bidid.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/no-imps.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/server-error-500.json create mode 100644 adapters/audienceNetwork/audienceNetworktest/supplemental/site-not-supported.json create mode 100644 adapters/between/between.go create mode 100644 adapters/between/between_test.go create mode 100644 adapters/between/betweentest/exemplary/multi-request.json create mode 100644 adapters/between/betweentest/exemplary/simple-site-banner.json create mode 100644 adapters/between/betweentest/params/race/banner.json create mode 100644 adapters/between/betweentest/supplemental/bad-bidder-ext.json create mode 100644 adapters/between/betweentest/supplemental/bad-dsp-request-example.json create mode 100644 adapters/between/betweentest/supplemental/bad-response-body.json create mode 100644 adapters/between/betweentest/supplemental/dsp-server-internal-error-example.json create mode 100644 adapters/between/betweentest/supplemental/missing-host.json create mode 100644 adapters/between/betweentest/supplemental/missing-imp.json create mode 100644 adapters/between/betweentest/supplemental/no-bids.json create mode 100644 adapters/between/betweentest/supplemental/unknown-status-code-example.json create mode 100644 adapters/between/betweentest/supplemental/zero-bid-request-error.json create mode 100644 adapters/brightroll/brightrolltest/supplemental/invalid-publisher.json create mode 100644 adapters/colossus/colossus.go create mode 100644 adapters/colossus/colossus_test.go create mode 100644 adapters/colossus/colossustest/exemplary/simple-banner.json create mode 100644 adapters/colossus/colossustest/exemplary/simple-video.json create mode 100644 adapters/colossus/colossustest/exemplary/simple-web-banner.json create mode 100644 adapters/colossus/colossustest/params/banner.json create mode 100644 adapters/colossus/colossustest/params/race/banner.json create mode 100644 adapters/colossus/colossustest/params/race/video.json create mode 100644 adapters/colossus/colossustest/params/video.json create mode 100644 adapters/colossus/colossustest/supplemental/bad-imp-ext.json create mode 100644 adapters/colossus/colossustest/supplemental/bad_response.json create mode 100644 adapters/colossus/colossustest/supplemental/bad_status_code.json create mode 100644 adapters/colossus/colossustest/supplemental/empty_imp_ext.json create mode 100644 adapters/colossus/colossustest/supplemental/imp_ext_empty_object.json create mode 100644 adapters/colossus/colossustest/supplemental/imp_ext_string.json create mode 100644 adapters/colossus/colossustest/supplemental/status-204.json create mode 100644 adapters/colossus/colossustest/supplemental/status-404.json create mode 100644 adapters/colossus/colossustest/supplemental/string_imp_ext.json create mode 100644 adapters/colossus/params_test.go create mode 100644 adapters/colossus/usersync.go create mode 100644 adapters/colossus/usersync_test.go create mode 100644 adapters/connectad/connectad.go create mode 100644 adapters/connectad/connectad_test.go create mode 100644 adapters/connectad/connectadtest/exemplary/optional-params.json create mode 100644 adapters/connectad/connectadtest/exemplary/simple-banner.json create mode 100644 adapters/connectad/connectadtest/params/race/banner.json create mode 100644 adapters/connectad/connectadtest/supplemental/204.json create mode 100644 adapters/connectad/connectadtest/supplemental/badresponse.json create mode 100644 adapters/connectad/connectadtest/supplemental/banner-multi.json create mode 100644 adapters/connectad/connectadtest/supplemental/err500.json create mode 100644 adapters/connectad/connectadtest/supplemental/ipv6.json create mode 100644 adapters/connectad/connectadtest/supplemental/no_banner.json create mode 100644 adapters/connectad/connectadtest/supplemental/no_device.json create mode 100644 adapters/connectad/connectadtest/supplemental/no_dnt.json create mode 100644 adapters/connectad/connectadtest/supplemental/no_ext.json create mode 100644 adapters/connectad/connectadtest/supplemental/no_format.json create mode 100644 adapters/connectad/connectadtest/supplemental/wrongext.json create mode 100644 adapters/connectad/params_test.go create mode 100644 adapters/connectad/usersync.go create mode 100644 adapters/connectad/usersync_test.go create mode 100644 adapters/conversant/cnvr_legacy.go create mode 100644 adapters/conversant/cnvr_legacy_test.go create mode 100644 adapters/conversant/conversanttest/exemplary/banner.json create mode 100644 adapters/conversant/conversanttest/exemplary/simple_app.json create mode 100644 adapters/conversant/conversanttest/exemplary/video.json create mode 100644 adapters/conversant/conversanttest/supplemental/missing_cnvrext.json create mode 100644 adapters/conversant/conversanttest/supplemental/missing_ext.json create mode 100644 adapters/conversant/conversanttest/supplemental/missing_siteid.json create mode 100644 adapters/conversant/conversanttest/supplemental/server_badresponse.json create mode 100644 adapters/conversant/conversanttest/supplemental/server_nocontent.json create mode 100644 adapters/conversant/conversanttest/supplemental/server_unknownstatus.json create mode 100644 adapters/dmx/dmxtest/exemplary/imp-populated-banner.json create mode 100644 adapters/emx_digital/emx_digitaltest/exemplary/banner-and-video-app.json create mode 100644 adapters/emx_digital/emx_digitaltest/exemplary/banner-and-video-site.json create mode 100644 adapters/emx_digital/emx_digitaltest/exemplary/banner-app.json create mode 100644 adapters/emx_digital/emx_digitaltest/exemplary/video-app.json create mode 100644 adapters/emx_digital/emx_digitaltest/exemplary/video-ctv.json create mode 100644 adapters/emx_digital/emx_digitaltest/exemplary/video-site.json create mode 100644 adapters/emx_digital/emx_digitaltest/params/race/video.json create mode 100644 adapters/emx_digital/emx_digitaltest/supplemental/app-domain-and-url-correctly-parsed.json create mode 100644 adapters/emx_digital/emx_digitaltest/supplemental/app-storeUrl-correctly-parsed.json create mode 100644 adapters/emx_digital/emx_digitaltest/supplemental/bad-imp-video-missing-mimes.json create mode 100644 adapters/emx_digital/emx_digitaltest/supplemental/bad-imp-video-missing-sizes.json create mode 100644 adapters/emx_digital/emx_digitaltest/supplemental/build-video-object.json create mode 100644 adapters/eplanning/eplanningtest/supplemental/bad-page-site.json create mode 100644 adapters/eplanning/eplanningtest/supplemental/site-page-and-url-correctly-parsed.json create mode 100644 adapters/gamma/gammatest/supplemental/missing-adm.json create mode 100644 adapters/gamma/gammatest/supplemental/nobid-signaling.json create mode 100644 adapters/gumgum/gumgumtest/exemplary/video.json create mode 100644 adapters/gumgum/gumgumtest/params/race/video.json create mode 100644 adapters/gumgum/gumgumtest/supplemental/missing-video-params.json create mode 100644 adapters/inmobi/inmobi.go create mode 100644 adapters/inmobi/inmobi_test.go create mode 100644 adapters/inmobi/inmobitest/exemplary/simple-banner.json create mode 100644 adapters/inmobi/inmobitest/exemplary/simple-video.json create mode 100644 adapters/inmobi/inmobitest/params/race/banner.json create mode 100644 adapters/inmobi/inmobitest/params/race/video.json create mode 100644 adapters/inmobi/inmobitest/supplemental/204.json create mode 100644 adapters/inmobi/inmobitest/supplemental/400.json create mode 100644 adapters/inmobi/inmobitest/supplemental/banner-format-coersion.json create mode 100644 adapters/inmobi/inmobitest/supplemental/ext-unmarshal-err.json create mode 100644 adapters/inmobi/inmobitest/supplemental/missing-plc-error.json create mode 100644 adapters/inmobi/inmobitest/supplemental/no-imp-error.json create mode 100644 adapters/invibes/invibes.go create mode 100644 adapters/invibes/invibes_test.go create mode 100644 adapters/invibes/invibestest/amp/amp-ad.json create mode 100644 adapters/invibes/invibestest/exemplary/advanced-ad.json create mode 100644 adapters/invibes/invibestest/exemplary/basic-ad.json create mode 100644 adapters/invibes/invibestest/exemplary/no-ad.json create mode 100644 adapters/invibes/invibestest/exemplary/test-ad.json create mode 100644 adapters/invibes/invibestest/params/race/banner.json create mode 100644 adapters/invibes/invibestest/supplemental/request-error-banner.json create mode 100644 adapters/invibes/invibestest/supplemental/request-error-impext.json create mode 100644 adapters/invibes/invibestest/supplemental/request-error-invibesparams.json create mode 100644 adapters/invibes/invibestest/supplemental/request-error-servererr.json create mode 100644 adapters/invibes/invibestest/supplemental/request-error-site.json create mode 100644 adapters/invibes/invibestest/supplemental/request-error-statuscode.json create mode 100644 adapters/invibes/params_test.go create mode 100644 adapters/invibes/usersync.go create mode 100644 adapters/invibes/usersync_test.go create mode 100644 adapters/krushmedia/krushmedia.go create mode 100644 adapters/krushmedia/krushmedia_test.go create mode 100644 adapters/krushmedia/krushmediatest/exemplary/banner-app.json create mode 100644 adapters/krushmedia/krushmediatest/exemplary/banner-web.json create mode 100644 adapters/krushmedia/krushmediatest/exemplary/native-app.json create mode 100644 adapters/krushmedia/krushmediatest/exemplary/native-web.json create mode 100644 adapters/krushmedia/krushmediatest/exemplary/video-app.json create mode 100644 adapters/krushmedia/krushmediatest/exemplary/video-web.json create mode 100644 adapters/krushmedia/krushmediatest/params/race/banner.json create mode 100644 adapters/krushmedia/krushmediatest/params/race/native.json create mode 100644 adapters/krushmedia/krushmediatest/params/race/video.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/invalid-ext-object.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/invalid-response.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/requires-imp-object.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/status-code-bad-request.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/status-code-no-content.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/status-code-other-error.json create mode 100644 adapters/krushmedia/krushmediatest/supplemental/status-code-service-unavailable.json create mode 100644 adapters/krushmedia/params_test.go create mode 100644 adapters/krushmedia/usersync.go create mode 100644 adapters/krushmedia/usersync_test.go create mode 100644 adapters/kubient/kubienttest/supplemental/missing-zoneid.json create mode 100644 adapters/kubient/kubienttest/supplemental/no-imps.json create mode 100644 adapters/logicad/logicad.go create mode 100644 adapters/logicad/logicad_test.go create mode 100644 adapters/logicad/logicadtest/exemplary/banner.json create mode 100644 adapters/logicad/logicadtest/params/race/banner.json create mode 100644 adapters/logicad/logicadtest/supplemental/checkImp.json create mode 100644 adapters/logicad/logicadtest/supplemental/ext.json create mode 100644 adapters/logicad/logicadtest/supplemental/missingtid.json create mode 100644 adapters/logicad/logicadtest/supplemental/multiImpSameTid.json create mode 100644 adapters/logicad/logicadtest/supplemental/responseCode.json create mode 100644 adapters/logicad/logicadtest/supplemental/responseNoBid.json create mode 100644 adapters/logicad/logicadtest/supplemental/responsebid.json create mode 100644 adapters/logicad/logicadtest/supplemental/site.json create mode 100644 adapters/logicad/params_test.go create mode 100644 adapters/logicad/usersync.go create mode 100644 adapters/logicad/usersync_test.go create mode 100644 adapters/nobid/nobid.go create mode 100644 adapters/nobid/nobid_test.go create mode 100644 adapters/nobid/nobidtest/exemplary/banner.json create mode 100644 adapters/nobid/nobidtest/supplemental/bad-mediatype.json create mode 100644 adapters/nobid/nobidtest/supplemental/bad-request.json create mode 100644 adapters/nobid/nobidtest/supplemental/bad-response.json create mode 100644 adapters/nobid/nobidtest/supplemental/missing-imps.json create mode 100644 adapters/nobid/nobidtest/supplemental/no-content.json create mode 100644 adapters/nobid/nobidtest/supplemental/notok-response.json create mode 100644 adapters/nobid/params_test.go create mode 100644 adapters/nobid/usersync.go create mode 100644 adapters/nobid/usersync_test.go create mode 100644 adapters/silvermob/params_test.go create mode 100644 adapters/silvermob/silvermob.go create mode 100644 adapters/silvermob/silvermob_test.go create mode 100644 adapters/silvermob/silvermobtest/exemplary/banner-app.json create mode 100644 adapters/silvermob/silvermobtest/exemplary/banner-multi-app.json create mode 100644 adapters/silvermob/silvermobtest/exemplary/native-app.json create mode 100644 adapters/silvermob/silvermobtest/exemplary/video-app.json create mode 100644 adapters/silvermob/silvermobtest/params/race/banner.json create mode 100644 adapters/silvermob/silvermobtest/params/race/native.json create mode 100644 adapters/silvermob/silvermobtest/params/race/video.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/empty-seatbid-array.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/invalid-response.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/invalid-silvermob-ext-object.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/status-code-bad-request.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/status-code-no-content.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/status-code-other-error.json create mode 100644 adapters/silvermob/silvermobtest/supplemental/status-code-service-unavailable.json create mode 100644 adapters/smaato/image.go create mode 100644 adapters/smaato/image_test.go create mode 100644 adapters/smaato/params_test.go create mode 100644 adapters/smaato/richmedia.go create mode 100644 adapters/smaato/richmedia_test.go create mode 100644 adapters/smaato/smaato.go create mode 100644 adapters/smaato/smaato_test.go create mode 100644 adapters/smaato/smaatotest/exemplary/simple-banner-richMedia.json create mode 100644 adapters/smaato/smaatotest/exemplary/simple-banner.json create mode 100644 adapters/smaato/smaatotest/exemplary/video.json create mode 100644 adapters/smaato/smaatotest/params/banner.json create mode 100644 adapters/smaato/smaatotest/supplemental/bad-adm-response.json create mode 100644 adapters/smaato/smaatotest/supplemental/bad-ext-req.json create mode 100644 adapters/smaato/smaatotest/supplemental/bad-imp-banner-format-req.json create mode 100644 adapters/smaato/smaatotest/supplemental/bad-user-ext-data-req.json create mode 100644 adapters/smaato/smaatotest/supplemental/bad-user-ext-req.json create mode 100644 adapters/smaato/smaatotest/supplemental/no-consent-info.json create mode 100644 adapters/smaato/smaatotest/supplemental/no-imp-req.json create mode 100644 adapters/smartadserver/params_test.go create mode 100644 adapters/smartadserver/smartadserver.go create mode 100644 adapters/smartadserver/smartadserver_test.go create mode 100644 adapters/smartadserver/smartadservertest/exemplary/multi-banner.json create mode 100644 adapters/smartadserver/smartadservertest/exemplary/simple-banner.json create mode 100644 adapters/smartadserver/smartadservertest/exemplary/simple-video.json create mode 100644 adapters/smartadserver/smartadservertest/params/race/banner.json create mode 100644 adapters/smartadserver/smartadservertest/params/race/video.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/request-no-bidder-object.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/request-no-ext-object.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/request-no-imp.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/request-site-recreated.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/response-200-without-body.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/response-204.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/response-400.json create mode 100644 adapters/smartadserver/smartadservertest/supplemental/response-500.json create mode 100644 adapters/smartadserver/usersync.go create mode 100644 adapters/smartadserver/usersync_test.go create mode 100644 adapters/smartyads/params_test.go create mode 100644 adapters/smartyads/smartyads.go create mode 100644 adapters/smartyads/smartyads_test.go create mode 100644 adapters/smartyads/smartyadstest/exemplary/banner-app.json create mode 100644 adapters/smartyads/smartyadstest/exemplary/banner-web.json create mode 100644 adapters/smartyads/smartyadstest/exemplary/native-app.json create mode 100644 adapters/smartyads/smartyadstest/exemplary/native-web.json create mode 100644 adapters/smartyads/smartyadstest/exemplary/video-app.json create mode 100644 adapters/smartyads/smartyadstest/exemplary/video-web.json create mode 100644 adapters/smartyads/smartyadstest/params/race/banner.json create mode 100644 adapters/smartyads/smartyadstest/params/race/native.json create mode 100644 adapters/smartyads/smartyadstest/params/race/video.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/empty-seatbid-array.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/invalid-response.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/invalid-smartyads-ext-object.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/status-code-bad-request.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/status-code-no-content.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/status-code-other-error.json create mode 100644 adapters/smartyads/smartyadstest/supplemental/status-code-service-unavailable.json create mode 100644 adapters/smartyads/usersync.go create mode 100644 adapters/smartyads/usersync_test.go create mode 100644 adapters/yieldmo/yieldmotest/exemplary/app-banner.json create mode 100644 adapters/yieldmo/yieldmotest/exemplary/app_video.json create mode 100644 adapters/yieldmo/yieldmotest/exemplary/simple_video.json create mode 100644 analytics/clients/http.go create mode 100644 analytics/event.go create mode 100644 analytics/pubstack/README.md create mode 100644 analytics/pubstack/config.go create mode 100644 analytics/pubstack/config_test.go create mode 100644 analytics/pubstack/eventchannel/eventchannel.go create mode 100644 analytics/pubstack/eventchannel/eventchannel_test.go create mode 100644 analytics/pubstack/eventchannel/sender.go create mode 100644 analytics/pubstack/eventchannel/sender_test.go create mode 100644 analytics/pubstack/helpers/json.go create mode 100644 analytics/pubstack/helpers/json_test.go create mode 100644 analytics/pubstack/mocks/mock_openrtb_request.json create mode 100644 analytics/pubstack/mocks/mock_openrtb_response.json create mode 100644 analytics/pubstack/pubstack_module.go create mode 100644 analytics/pubstack/pubstack_module_test.go create mode 100644 config/accounts.go create mode 100644 config/accounts_test.go create mode 100644 config/requestvalidation.go create mode 100644 config/requestvalidation_test.go create mode 100644 devcontainer.md delete mode 100644 docs/bidders/adtarget.md delete mode 100644 docs/bidders/appnexus.md delete mode 100644 docs/bidders/audienceNetwork.md delete mode 100644 docs/bidders/avocet.md delete mode 100644 docs/bidders/beachfront.md delete mode 100644 docs/bidders/emx_digital.md delete mode 100644 docs/bidders/kidoz.md delete mode 100644 docs/bidders/openx.md delete mode 100644 docs/bidders/pubmatic.md delete mode 100644 docs/bidders/pubnative.md delete mode 100644 docs/bidders/rubicon.md delete mode 100644 docs/bidders/smartrtb.md delete mode 100644 docs/bidders/sovrn.md delete mode 100644 docs/bidders/tappx.md delete mode 100644 docs/developers/Prebid Server Event Notifications - Tech Spec.pdf delete mode 100644 docs/developers/add-new-analytics-module.md delete mode 100644 docs/developers/currency-converter.md create mode 100644 docs/developers/features.md delete mode 100644 docs/developers/gdpr.md create mode 100644 docs/endpoints.md delete mode 100644 docs/endpoints/bidders/params.md delete mode 100644 docs/endpoints/cookieSync.md delete mode 100644 docs/endpoints/currency_rates.md delete mode 100644 docs/endpoints/info/bidders.md delete mode 100644 docs/endpoints/info/bidders/bidderName.md delete mode 100644 docs/endpoints/openrtb2/amp.md delete mode 100644 docs/endpoints/openrtb2/auction.md delete mode 100644 docs/endpoints/setuid.md delete mode 100644 docs/endpoints/status.md create mode 100644 endpoints/events/account_test.go create mode 100644 endpoints/events/event.go create mode 100644 endpoints/events/event_test.go create mode 100644 endpoints/events/vtrack.go create mode 100644 endpoints/events/vtrack_test.go create mode 100644 endpoints/openrtb2/sample-requests/account-required/no-account/not-required-no-acct.json create mode 100644 endpoints/openrtb2/sample-requests/account-required/no-account/required-no-acct.json delete mode 100644 endpoints/openrtb2/sample-requests/account-required/no-acct.json rename endpoints/openrtb2/sample-requests/account-required/{with-acct.json => valid-acct.json} (79%) create mode 100644 endpoints/openrtb2/sample-requests/account-required/with-account/required-blacklisted-acct.json create mode 100644 endpoints/openrtb2/sample-requests/account-required/with-account/required-with-acct.json create mode 100644 endpoints/openrtb2/sample-requests/aliased/multiple-alias.json delete mode 100644 endpoints/openrtb2/sample-requests/blacklisted/blacklisted-acct.json create mode 100644 endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app-publisher.json create mode 100644 endpoints/openrtb2/sample-requests/blacklisted/blacklisted-site-publisher.json create mode 100644 endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-ext-bidder.json create mode 100644 endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-prebid-bidder.json create mode 100644 endpoints/openrtb2/sample-requests/invalid-native/asset-title-empty.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-native/asset-title-no-length.json create mode 100644 endpoints/openrtb2/sample-requests/invalid-native/eventtracker-event-large.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-native/eventtracker-type-large.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_2.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/app-bad-ext.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/array.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/banner-h-zero.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/banner-null.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/banner-w-zero.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/boolean.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/float.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/integer.json rename endpoints/openrtb2/sample-requests/{aliased/site.json => invalid-whole/invalid-source.json} (53%) create mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/malformed-bid-request.json delete mode 100644 endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-badtype.json rename endpoints/openrtb2/sample-requests/invalid-whole/{user-gdpr-invalid.json => user-gdpr-consent-invalid.json} (70%) delete mode 100644 endpoints/openrtb2/sample-requests/valid-native/sample-v1.1.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-native/sample-v1.2.json create mode 100644 endpoints/openrtb2/sample-requests/valid-native/video-asset-event-tracker.json create mode 100644 endpoints/openrtb2/sample-requests/valid-native/with-video-asset.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/app.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/bid-adjustments.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-bids.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-vast.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/ccpa-invalid.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-device-only.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-no-extension.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial.json create mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-dnt.json create mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv4.json create mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv6.json delete mode 100644 endpoints/openrtb2/sample-requests/valid-whole/supplementary/timeout.json create mode 100644 endpoints/openrtb2/sample-requests/video/video_valid_sample_appendbiddernames.json create mode 100644 exchange/cachetest/debuglog_enabled_no_winners_nor_bids.json create mode 100644 exchange/exchangetest/append-bidder-names.json create mode 100644 exchange/exchangetest/ccpa-nosale-any-bidder.json create mode 100644 exchange/exchangetest/ccpa-nosale-specific-bidder.json create mode 100644 exchange/exchangetest/debuglog_enabled_no_bids.json create mode 100644 exchange/exchangetest/firstpartydata-imp-ext-multiple-bidders.json create mode 100644 exchange/exchangetest/firstpartydata-imp-ext-multiple-prebid-bidders.json create mode 100644 exchange/exchangetest/firstpartydata-imp-ext-one-bidder.json create mode 100644 exchange/exchangetest/firstpartydata-imp-ext-one-prebid-bidder.json create mode 100644 exchange/exchangetest/gdpr-geo-eu-off-device.json create mode 100644 exchange/exchangetest/gdpr-geo-eu-off.json create mode 100644 exchange/exchangetest/gdpr-geo-eu-on-featureflag-off.json create mode 100644 exchange/exchangetest/gdpr-geo-eu-on.json create mode 100644 exchange/exchangetest/gdpr-geo-usa-off.json create mode 100644 exchange/exchangetest/gdpr-geo-usa-on.json create mode 100644 exchange/impcustomcachekeytest/multiImpVideo.json create mode 100644 exchange/impcustomcachekeytest/multiImpVideoNoIncludeBidderKeys.json create mode 100644 openrtb_ext/deal_tier.go create mode 100644 openrtb_ext/deal_tier_test.go create mode 100644 openrtb_ext/imp_acuityads.go create mode 100644 openrtb_ext/imp_adman.go create mode 100644 openrtb_ext/imp_adprime.go create mode 100644 openrtb_ext/imp_amx.go create mode 100644 openrtb_ext/imp_between.go create mode 100644 openrtb_ext/imp_colossus.go create mode 100644 openrtb_ext/imp_connectad.go create mode 100644 openrtb_ext/imp_conversant.go create mode 100644 openrtb_ext/imp_inmobi.go create mode 100644 openrtb_ext/imp_invibes.go create mode 100644 openrtb_ext/imp_krushmedia.go create mode 100644 openrtb_ext/imp_kubient.go create mode 100644 openrtb_ext/imp_logicad.go create mode 100644 openrtb_ext/imp_nobid.go create mode 100644 openrtb_ext/imp_silvermob.go create mode 100644 openrtb_ext/imp_smaato.go create mode 100644 openrtb_ext/imp_smartadserver.go create mode 100644 openrtb_ext/imp_smartyads.go delete mode 100644 prebid/prebid.go create mode 100644 privacy/ccpa/consentwriter.go create mode 100644 privacy/ccpa/consentwriter_test.go create mode 100644 privacy/ccpa/parsedpolicy.go create mode 100644 privacy/ccpa/parsedpolicy_test.go create mode 100644 privacy/enforcer.go create mode 100644 privacy/enforcer_test.go create mode 100644 privacy/gdpr/consentwriter.go create mode 100644 privacy/gdpr/consentwriter_test.go delete mode 100644 privacy/policies_test.go create mode 100644 privacy/writer.go create mode 100644 privacy/writer_test.go create mode 100644 static/bidder-info/acuityads.yaml create mode 100644 static/bidder-info/adman.yaml create mode 100644 static/bidder-info/adprime.yaml create mode 100644 static/bidder-info/amx.yaml create mode 100644 static/bidder-info/between.yaml create mode 100644 static/bidder-info/colossus.yaml create mode 100644 static/bidder-info/connectad.yaml create mode 100644 static/bidder-info/inmobi.yaml create mode 100644 static/bidder-info/invibes.yaml create mode 100644 static/bidder-info/krushmedia.yaml create mode 100644 static/bidder-info/logicad.yaml create mode 100644 static/bidder-info/nobid.yaml create mode 100644 static/bidder-info/silvermob.yaml create mode 100644 static/bidder-info/smaato.yaml create mode 100644 static/bidder-info/smartadserver.yaml create mode 100644 static/bidder-info/smartyads.yaml create mode 100644 static/bidder-params/acuityads.json create mode 100644 static/bidder-params/adman.json create mode 100644 static/bidder-params/adprime.json create mode 100644 static/bidder-params/amx.json create mode 100644 static/bidder-params/between.json create mode 100644 static/bidder-params/colossus.json create mode 100644 static/bidder-params/connectad.json create mode 100644 static/bidder-params/inmobi.json create mode 100644 static/bidder-params/invibes.json create mode 100644 static/bidder-params/krushmedia.json create mode 100644 static/bidder-params/logicad.json create mode 100644 static/bidder-params/nobid.json create mode 100644 static/bidder-params/silvermob.json create mode 100644 static/bidder-params/smaato.json create mode 100644 static/bidder-params/smartadserver.json create mode 100644 static/bidder-params/smartyads.json create mode 100644 static/tcf1/fallback_gvl.json create mode 100644 stored_requests/backends/file_fetcher/test/accounts/valid.json create mode 100644 stored_requests/data/by_id/accounts/test.json create mode 100644 stored_requests/events/postgres/database.go create mode 100644 stored_requests/events/postgres/database_test.go delete mode 100644 stored_requests/events/postgres/polling.go delete mode 100644 stored_requests/events/postgres/polling_test.go delete mode 100644 stored_requests/events/postgres/startup.go delete mode 100644 stored_requests/events/postgres/startup_test.go create mode 100644 util/httputil/httputil.go create mode 100644 util/httputil/httputil_test.go create mode 100644 util/iputil/parse.go create mode 100644 util/iputil/parse_test.go create mode 100644 util/iputil/validator.go create mode 100644 util/iputil/validator_test.go create mode 100644 util/maputil/maputil.go create mode 100644 util/maputil/maputil_test.go create mode 100644 util/task/ticker_task.go create mode 100644 util/task/ticker_task_test.go create mode 100644 util/timeutil/time.go diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile new file mode 100644 index 00000000000..d8373fd4c57 --- /dev/null +++ b/.devcontainer/Dockerfile @@ -0,0 +1,18 @@ +# From https://github.com/microsoft/vscode-dev-containers/blob/master/containers/go/.devcontainer/Dockerfile +ARG VARIANT=1 +FROM mcr.microsoft.com/vscode/devcontainers/go:${VARIANT} + +# [Optional] Install a version of Node.js using nvm for front end dev +ARG INSTALL_NODE="true" +ARG NODE_VERSION="lts/*" +RUN if [ "${INSTALL_NODE}" = "true" ]; then su vscode -c "source /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi + +# [Optional] Uncomment this section to install additional OS packages. +RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \ + && apt-get -y install --no-install-recommends vim + +# [Optional] Uncomment the next line to use go get to install anything else you need +# RUN go get -x + +# [Optional] Uncomment this line to install global node packages. +# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g " 2>&1 diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json new file mode 100644 index 00000000000..b2c53776ad4 --- /dev/null +++ b/.devcontainer/devcontainer.json @@ -0,0 +1,45 @@ +// For format details, see https://aka.ms/vscode-remote/devcontainer.json or this file's README at: +// https://github.com/microsoft/vscode-dev-containers/tree/v0.112.0/containers/go +{ + "name": "Go", + "build": { + "dockerfile": "Dockerfile", + "args": { + // Update the VARIANT arg to pick a version of Go: 1, 1.15, 1.14 + "VARIANT": "1.14", + // Options + "INSTALL_NODE": "false", + "NODE_VERSION": "lts/*", + } + }, + "containerEnv": { + "GOPRIVATE": "${localEnv:GOPRIVATE}", + }, + "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ], + + // Set *default* container specific settings.json values on container create. + "settings": { + "terminal.integrated.shell.linux": "/bin/bash", + "go.useGoProxyToCheckForToolUpdates": false, + "go.gopath": "/go", + //"go.toolsGopath": "/tmp/go", + }, + + // Add the IDs of extensions you want installed when the container is created. + "extensions": [ + "golang.Go", + "ms-azuretools.vscode-docker", + "redhat.vscode-xml", + "redhat.vscode-yaml", + "eamodio.gitlens", + ], + + // Use 'forwardPorts' to make a list of ports inside the container available locally. + "forwardPorts": [8000,8001,6060], + + // Use 'postCreateCommand' to run commands after the container is created. + "postCreateCommand": "mkdir ~/.ssh; ssh-keyscan github.com > ~/.ssh/known_hosts", + + // Uncomment to connect as a non-root user. See https://aka.ms/vscode-remote/containers/non-root. + "remoteUser": "vscode" +} diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml new file mode 100644 index 00000000000..d7bb50fbabf --- /dev/null +++ b/.github/workflows/validate.yml @@ -0,0 +1,28 @@ +on: + push: + branches: + - master + pull_request: + release: + types: + - created +name: Validate +jobs: + Go: + strategy: + matrix: + go-version: [1.13.x, 1.14.x, 1.15.x] + os: [ubuntu-18.04] + runs-on: ${{ matrix.os }} + steps: + - name: Install Go + uses: actions/setup-go@v2 + with: + go-version: ${{ matrix.go-version }} + - name: Checkout code + uses: actions/checkout@v2 + - name: Validate + run: | + ./validate.sh --nofmt --cov --race 10 + env: + GO111MODULE: "on" diff --git a/.gitignore b/.gitignore index 60c24e79c0d..79076f9be84 100644 --- a/.gitignore +++ b/.gitignore @@ -8,7 +8,6 @@ _obj _test .cover/ .idea/ -.vscode/ # Architecture specific extensions/prefixes *.[568vq] @@ -46,6 +45,7 @@ analytics/filesystem/testFiles/ # static/version.txt .idea/ +.vscode/ # autogenerated mac file diff --git a/README.md b/README.md index b3c795bf803..145883587fd 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,13 @@ It is managed by [Prebid.org](http://prebid.org/overview/what-is-prebid-org.html and upholds the principles from the [Prebid Code of Conduct](http://prebid.org/wrapper_code_of_conduct.html). This project does not support the same set of Bidders as Prebid.js, although there is overlap. -The current set can be found in the [adapters](./adapters) package. If you don't see the one you want, feel free to [contribute it](docs/developers/add-new-bidder.md). +The current set can be found in the [adapters](./adapters) package. If you don't see the one you want, feel free to [contribute it](https://docs.prebid.org/prebid-server/developers/add-new-bidder-go.html). For more information, see: -- [What is Prebid?](http://prebid.org/overview/intro.html) -- [Getting started with Prebid Server](http://prebid.org/dev-docs/get-started-with-prebid-server.html) -- [Current Bidders](http://prebid.org/dev-docs/prebid-server-bidders.html) +- [What is Prebid?](https://prebid.org/overview/intro.html) +- [Prebid Server Overview](https://docs.prebid.org/prebid-server/overview/prebid-server-overview.html) +- [Current Bidders](http://prebid.org/dev-docs/pbs-bidders.html) ## Installation @@ -45,15 +45,18 @@ go build . ``` Load the landing page in your browser at `http://localhost:8000/`. -For the full API reference, see [docs/endpoints](docs/endpoints) +For the full API reference, see [the endpoint documentation](https://docs.prebid.org/prebid-server/endpoints/pbs-endpoint-overview.html) ## Contributing -Want to [add an adapter](docs/developers/add-new-bidder.md)? Found a bug? Great! -This project is in its infancy, and many things can be improved. - +Want to [add an adapter](https://docs.prebid.org/prebid-server/developers/add-new-bidder-go.html)? Found a bug? Great! Report bugs, request features, and suggest improvements [on Github](https://github.com/PubMatic-OpenWrap/prebid-server/issues). Or better yet, [open a pull request](https://github.com/PubMatic-OpenWrap/prebid-server/compare) with the changes you'd like to see. + +## IDE Setup for PBS-Go development + +The quickest way to start developing PBS-Go in a reproducible environment isolated from your host OS +is by using this [VScode Remote Container Setup](devcontainer.md) diff --git a/account/account.go b/account/account.go new file mode 100644 index 00000000000..43ba806a6da --- /dev/null +++ b/account/account.go @@ -0,0 +1,69 @@ +package account + +import ( + "context" + "encoding/json" + "fmt" + + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + jsonpatch "github.com/evanphx/json-patch" +) + +// GetAccount looks up the config.Account object referenced by the given accountID, with access rules applied +func GetAccount(ctx context.Context, cfg *config.Configuration, fetcher stored_requests.AccountFetcher, accountID string) (account *config.Account, errs []error) { + // Check BlacklistedAcctMap until we have deprecated it + if _, found := cfg.BlacklistedAcctMap[accountID]; found { + return nil, []error{&errortypes.BlacklistedAcct{ + Message: fmt.Sprintf("Prebid-server has disabled Account ID: %s, please reach out to the prebid server host.", accountID), + }} + } + if cfg.AccountRequired && accountID == pbsmetrics.PublisherUnknown { + return nil, []error{&errortypes.AcctRequired{ + Message: fmt.Sprintf("Prebid-server has been configured to discard requests without a valid Account ID. Please reach out to the prebid server host."), + }} + } + if accountJSON, accErrs := fetcher.FetchAccount(ctx, accountID); len(accErrs) > 0 || accountJSON == nil { + // accountID does not reference a valid account + for _, e := range accErrs { + if _, ok := e.(stored_requests.NotFoundError); !ok { + errs = append(errs, e) + } + } + if cfg.AccountRequired && cfg.AccountDefaults.Disabled { + errs = append(errs, &errortypes.AcctRequired{ + Message: fmt.Sprintf("Prebid-server could not verify the Account ID. Please reach out to the prebid server host."), + }) + return nil, errs + } + // Make a copy of AccountDefaults instead of taking a reference, + // to preserve original accountID in case is needed to check NonStandardPublisherMap + pubAccount := cfg.AccountDefaults + pubAccount.ID = accountID + account = &pubAccount + } else { + // accountID resolved to a valid account, merge with AccountDefaults for a complete config + account = &config.Account{} + completeJSON, err := jsonpatch.MergePatch(cfg.AccountDefaultsJSON(), accountJSON) + if err == nil { + err = json.Unmarshal(completeJSON, account) + } + if err != nil { + errs = append(errs, err) + return nil, errs + } + // Fill in ID if needed, so it can be left out of account definition + if len(account.ID) == 0 { + account.ID = accountID + } + } + if account.Disabled { + errs = append(errs, &errortypes.BlacklistedAcct{ + Message: fmt.Sprintf("Prebid-server has disabled Account ID: %s, please reach out to the prebid server host.", accountID), + }) + return nil, errs + } + return account, nil +} diff --git a/account/account_test.go b/account/account_test.go new file mode 100644 index 00000000000..e8acdfe0f5f --- /dev/null +++ b/account/account_test.go @@ -0,0 +1,94 @@ +package account + +import ( + "context" + "encoding/json" + "fmt" + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/stretchr/testify/assert" +) + +var mockAccountData = map[string]json.RawMessage{ + "valid_acct": json.RawMessage(`{"disabled":false}`), + "disabled_acct": json.RawMessage(`{"disabled":true}`), +} + +type mockAccountFetcher struct { +} + +func (af mockAccountFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + if account, ok := mockAccountData[accountID]; ok { + return account, nil + } + return nil, []error{stored_requests.NotFoundError{accountID, "Account"}} +} + +func TestGetAccount(t *testing.T) { + unknown := pbsmetrics.PublisherUnknown + testCases := []struct { + accountID string + // account_required + required bool + // account_defaults.disabled + disabled bool + // expected error, or nil if account should be found + err error + }{ + // Blacklisted account is always rejected even in permissive setup + {accountID: "bad_acct", required: false, disabled: false, err: &errortypes.BlacklistedAcct{}}, + + // empty pubID + {accountID: unknown, required: false, disabled: false, err: nil}, + {accountID: unknown, required: true, disabled: false, err: &errortypes.AcctRequired{}}, + {accountID: unknown, required: false, disabled: true, err: &errortypes.BlacklistedAcct{}}, + {accountID: unknown, required: true, disabled: true, err: &errortypes.AcctRequired{}}, + + // pubID given but is not a valid host account (does not exist) + {accountID: "doesnt_exist_acct", required: false, disabled: false, err: nil}, + {accountID: "doesnt_exist_acct", required: true, disabled: false, err: nil}, + {accountID: "doesnt_exist_acct", required: false, disabled: true, err: &errortypes.BlacklistedAcct{}}, + {accountID: "doesnt_exist_acct", required: true, disabled: true, err: &errortypes.AcctRequired{}}, + + // pubID given and matches a valid host account with Disabled: false + {accountID: "valid_acct", required: false, disabled: false, err: nil}, + {accountID: "valid_acct", required: true, disabled: false, err: nil}, + {accountID: "valid_acct", required: false, disabled: true, err: nil}, + {accountID: "valid_acct", required: true, disabled: true, err: nil}, + + // pubID given and matches a host account explicitly disabled (Disabled: true on account json) + {accountID: "disabled_acct", required: false, disabled: false, err: &errortypes.BlacklistedAcct{}}, + {accountID: "disabled_acct", required: true, disabled: false, err: &errortypes.BlacklistedAcct{}}, + {accountID: "disabled_acct", required: false, disabled: true, err: &errortypes.BlacklistedAcct{}}, + {accountID: "disabled_acct", required: true, disabled: true, err: &errortypes.BlacklistedAcct{}}, + } + + for _, test := range testCases { + description := fmt.Sprintf(`ID=%s/required=%t/disabled=%t`, test.accountID, test.required, test.disabled) + t.Run(description, func(t *testing.T) { + cfg := &config.Configuration{ + BlacklistedAcctMap: map[string]bool{"bad_acct": true}, + AccountRequired: test.required, + AccountDefaults: config.Account{Disabled: test.disabled}, + } + fetcher := &mockAccountFetcher{} + assert.NoError(t, cfg.MarshalAccountDefaults()) + + account, errors := GetAccount(context.Background(), cfg, fetcher, test.accountID) + + if test.err == nil { + assert.Empty(t, errors) + assert.Equal(t, test.accountID, account.ID, "account.ID must match requested ID") + assert.Equal(t, false, account.Disabled, "returned account must not be disabled") + } else { + assert.NotEmpty(t, errors, "expected errors but got success") + assert.Nil(t, account, "return account must be nil on error") + assert.IsType(t, test.err, errors[0], "error is of unexpected type") + } + }) + } +} diff --git a/adapters/33across/33across.go b/adapters/33across/33across.go index a12e00ce544..9b622e4f94e 100644 --- a/adapters/33across/33across.go +++ b/adapters/33across/33across.go @@ -24,6 +24,14 @@ type ext struct { Zoneid string `json:"zoneid,omitempty"` } +type bidExt struct { + Ttx bidTtxExt `json:"ttx,omitempty"` +} + +type bidTtxExt struct { + MediaType string `json:mediaType,omitempty` +} + // MakeRequests create the object for TTX Reqeust. func (a *TtxAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *adapters.ExtraRequestInfo) ([]*adapters.RequestData, []error) { var errs []error @@ -49,6 +57,14 @@ func (a *TtxAdapter) makeRequest(request *openrtb.BidRequest) (*adapters.Request errs = append(errs, err) } + if reqCopy.Imp[0].Banner == nil && reqCopy.Imp[0].Video == nil { + errs = append(errs, &errortypes.BadInput{ + Message: "At least one of [banner, video] formats must be defined in Imp. None found", + }) + + return nil, errs + } + // Last Step reqJSON, err := json.Marshal(reqCopy) if err != nil { @@ -104,6 +120,19 @@ func preprocess(request *openrtb.BidRequest) error { siteCopy.ID = ttxExt.SiteId request.Site = &siteCopy + // Validate Video if it exists + if imp.Video != nil { + videoCopy, err := validateVideoParams(imp.Video, impExt.Ttx.Prod) + + imp.Video = videoCopy + + if err != nil { + return &errortypes.BadInput{ + Message: err.Error(), + } + } + } + return nil } @@ -135,9 +164,18 @@ func (a *TtxAdapter) MakeBids(internalRequest *openrtb.BidRequest, externalReque for _, sb := range bidResp.SeatBid { for i := range sb.Bid { + var bidExt bidExt + var bidType openrtb_ext.BidType + + if err := json.Unmarshal(sb.Bid[i].Ext, &bidExt); err != nil { + bidType = openrtb_ext.BidTypeBanner + } else { + bidType = getBidType(bidExt) + } + bidResponse.Bids = append(bidResponse.Bids, &adapters.TypedBid{ Bid: &sb.Bid[i], - BidType: "banner", + BidType: bidType, }) } } @@ -145,6 +183,43 @@ func (a *TtxAdapter) MakeBids(internalRequest *openrtb.BidRequest, externalReque } +func validateVideoParams(video *openrtb.Video, prod string) (*openrtb.Video, error) { + videoCopy := video + if videoCopy.W == 0 || + videoCopy.H == 0 || + videoCopy.Protocols == nil || + videoCopy.MIMEs == nil || + videoCopy.PlaybackMethod == nil { + + return nil, &errortypes.BadInput{ + Message: "One or more invalid or missing video field(s) w, h, protocols, mimes, playbackmethod", + } + } + + if videoCopy.Placement == 0 { + videoCopy.Placement = 2 + } + + if prod == "instream" { + videoCopy.Placement = 1 + + if videoCopy.StartDelay == nil { + videoCopy.StartDelay = openrtb.StartDelay.Ptr(0) + } + } + + return videoCopy, nil +} + +func getBidType(ext bidExt) openrtb_ext.BidType { + if ext.Ttx.MediaType == "video" { + return openrtb_ext.BidTypeVideo + } + + return openrtb_ext.BidTypeBanner +} + +// New33AcrossBidder configures bidder endpoint func New33AcrossBidder(endpoint string) *TtxAdapter { return &TtxAdapter{ endpoint: endpoint, diff --git a/adapters/33across/33across_test.go b/adapters/33across/33across_test.go index 1ec04dacb9e..efb771c6385 100644 --- a/adapters/33across/33across_test.go +++ b/adapters/33across/33across_test.go @@ -7,5 +7,5 @@ import ( ) func TestJsonSamples(t *testing.T) { - adapterstest.RunJSONBidderTest(t, "33across", New33AcrossBidder("http://ssc.33across.com")) + adapterstest.RunJSONBidderTest(t, "33acrosstest", New33AcrossBidder("http://ssc.33across.com")) } diff --git a/adapters/33across/33acrosstest/exemplary/bidresponse-defaults.json b/adapters/33across/33acrosstest/exemplary/bidresponse-defaults.json new file mode 100644 index 00000000000..bb0e6585fd0 --- /dev/null +++ b/adapters/33across/33acrosstest/exemplary/bidresponse-defaults.json @@ -0,0 +1,100 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 1, + "startdelay": -2, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "bidder": { + "siteId": "fake-site-id", + "productId": "instream" + } + } + } + ], + "site": {} + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://ssc.33across.com", + "body": { + "id": "test-request-id", + "imp": [ + { + "id":"test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 1, + "startdelay": -2, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "ttx": { + "prod": "instream" + } + } + } + ], + "site": { + "id": "fake-site-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "seat": "ttx", + "bid": [{ + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.500000, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "h": 90, + "w": 728 + }] + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.5, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "w": 728, + "h": 90 + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/33across/33acrosstest/exemplary/instream-video-defaults.json b/adapters/33across/33acrosstest/exemplary/instream-video-defaults.json new file mode 100644 index 00000000000..479b197077a --- /dev/null +++ b/adapters/33across/33acrosstest/exemplary/instream-video-defaults.json @@ -0,0 +1,108 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "bidder": { + "siteId": "fake-site-id", + "productId": "instream" + } + } + } + ], + "site": {} + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://ssc.33across.com", + "body": { + "id": "test-request-id", + "imp": [ + { + "id":"test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 1, + "startdelay": 0, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "ttx": { + "prod": "instream" + } + } + } + ], + "site": { + "id": "fake-site-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "seat": "ttx", + "bid": [{ + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.500000, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "h": 90, + "w": 728, + "ext": { + "ttx": { + "mediaType": "video" + } + } + }] + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.5, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "w": 728, + "h": 90, + "ext": { + "ttx": { + "mediaType": "video" + } + } + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/33across/33acrosstest/exemplary/multi-format.json b/adapters/33across/33acrosstest/exemplary/multi-format.json new file mode 100644 index 00000000000..db15955ca87 --- /dev/null +++ b/adapters/33across/33acrosstest/exemplary/multi-format.json @@ -0,0 +1,103 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [{"w": 728, "h": 90}] + }, + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "bidder": { + "siteId": "fake-site-id", + "productId": "inview" + } + } + } + ], + "site": {} + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://ssc.33across.com", + "body": { + "id": "test-request-id", + "imp": [ + { + "id":"test-imp-id", + "banner": { + "format": [{"w": 728, "h": 90}] + }, + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 2, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "ttx": { + "prod": "inview" + } + } + } + ], + "site": { + "id": "fake-site-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "seat": "ttx", + "bid": [{ + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.500000, + "adm": "some-test-ad", + "crid": "crid_10", + "h": 90, + "w": 728 + }] + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.5, + "adm": "some-test-ad", + "crid": "crid_10", + "w": 728, + "h": 90 + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/33across/33across/exemplary/optional-params.json b/adapters/33across/33acrosstest/exemplary/optional-params.json similarity index 100% rename from adapters/33across/33across/exemplary/optional-params.json rename to adapters/33across/33acrosstest/exemplary/optional-params.json diff --git a/adapters/33across/33acrosstest/exemplary/outstream-video-defaults.json b/adapters/33across/33acrosstest/exemplary/outstream-video-defaults.json new file mode 100644 index 00000000000..c0c31168684 --- /dev/null +++ b/adapters/33across/33acrosstest/exemplary/outstream-video-defaults.json @@ -0,0 +1,107 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "bidder": { + "siteId": "fake-site-id", + "productId": "siab" + } + } + } + ], + "site": {} + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://ssc.33across.com", + "body": { + "id": "test-request-id", + "imp": [ + { + "id":"test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 2, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "ttx": { + "prod": "siab" + } + } + } + ], + "site": { + "id": "fake-site-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "seat": "ttx", + "bid": [{ + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.500000, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "h": 90, + "w": 728, + "ext": { + "ttx": { + "mediaType": "video" + } + } + }] + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.5, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "w": 728, + "h": 90, + "ext": { + "ttx": { + "mediaType": "video" + } + } + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/33across/33across/exemplary/simple-banner.json b/adapters/33across/33acrosstest/exemplary/simple-banner.json similarity index 85% rename from adapters/33across/33across/exemplary/simple-banner.json rename to adapters/33across/33acrosstest/exemplary/simple-banner.json index 074badade07..d8c215c06ae 100644 --- a/adapters/33across/33across/exemplary/simple-banner.json +++ b/adapters/33across/33acrosstest/exemplary/simple-banner.json @@ -56,7 +56,12 @@ "adm": "some-test-ad", "crid": "crid_10", "h": 90, - "w": 728 + "w": 728, + "ext": { + "ttx": { + "mediaType": "banner" + } + } }] } ], @@ -78,7 +83,12 @@ "adm": "some-test-ad", "crid": "crid_10", "w": 728, - "h": 90 + "h": 90, + "ext": { + "ttx": { + "mediaType": "banner" + } + } }, "type": "banner" } diff --git a/adapters/33across/33acrosstest/exemplary/simple-video.json b/adapters/33across/33acrosstest/exemplary/simple-video.json new file mode 100644 index 00000000000..55337b92827 --- /dev/null +++ b/adapters/33across/33acrosstest/exemplary/simple-video.json @@ -0,0 +1,110 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 1, + "startdelay": -2, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "bidder": { + "siteId": "fake-site-id", + "productId": "instream" + } + } + } + ], + "site": {} + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://ssc.33across.com", + "body": { + "id": "test-request-id", + "imp": [ + { + "id":"test-imp-id", + "video": { + "w": 728, + "h": 90, + "protocols": [2], + "placement": 1, + "startdelay": -2, + "playbackmethod": [2], + "mimes": ["foo", "bar"] + }, + "ext": { + "ttx": { + "prod": "instream" + } + } + } + ], + "site": { + "id": "fake-site-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "seat": "ttx", + "bid": [{ + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.500000, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "h": 90, + "w": 728, + "ext": { + "ttx": { + "mediaType": "video" + } + } + }] + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "8ee514f1-b2b8-4abb-89fd-084437d1e800", + "impid": "test-imp-id", + "price": 0.5, + "adm": "some-test-vast-ad", + "crid": "crid_10", + "w": 728, + "h": 90, + "ext": { + "ttx": { + "mediaType": "video" + } + } + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/33across/33across/params/race/banner.json b/adapters/33across/33acrosstest/params/race/banner.json similarity index 100% rename from adapters/33across/33across/params/race/banner.json rename to adapters/33across/33acrosstest/params/race/banner.json diff --git a/adapters/33across/33acrosstest/params/race/video.json b/adapters/33across/33acrosstest/params/race/video.json new file mode 100644 index 00000000000..9df849ad94b --- /dev/null +++ b/adapters/33across/33acrosstest/params/race/video.json @@ -0,0 +1,6 @@ +{ + "productId": "siab", + "siteId": "33across", + "zoneId": "33AcrossZone" + } + \ No newline at end of file diff --git a/adapters/33across/33acrosstest/supplemental/status-not-ok.json b/adapters/33across/33acrosstest/supplemental/status-not-ok.json new file mode 100644 index 00000000000..98fe01c2e50 --- /dev/null +++ b/adapters/33across/33acrosstest/supplemental/status-not-ok.json @@ -0,0 +1,67 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [{"w": 728, "h": 90}] + }, + "ext": { + "bidder": { + "siteId": "fake-invalid-site-id", + "productId": "inview" + } + } + } + ], + "site": {} + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://ssc.33across.com", + "body": { + "id": "test-request-id", + "imp": [ + { + "id":"test-imp-id", + "banner": { + "format": [{"w": 728, "h": 90}] + }, + "ext": { + "ttx": { + "prod": "inview" + } + } + } + ], + "site": { + "id": "fake-invalid-site-id" + } + } + }, + "mockResponse": { + "status": 400, + "body": { + "error": { + "message": "Validation failed", + "details": [ + { + "message": "site.id is invalid" + } + ] + } + } + } + } + ], + + "expectedMakeBidsErrors": [ + { + "value": "Unexpected status code: 400. Run with request.debug = 1 for more info", + "comparison": "literal" + } + ] +} diff --git a/adapters/33across/33acrosstest/supplemental/video-validation-fail.json b/adapters/33across/33acrosstest/supplemental/video-validation-fail.json new file mode 100644 index 00000000000..97cb79bd26c --- /dev/null +++ b/adapters/33across/33acrosstest/supplemental/video-validation-fail.json @@ -0,0 +1,32 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "video": { + "w": 728, + "h": 90 + }, + "ext": { + "bidder": { + "siteId": "fake-site-id", + "productId": "siab" + } + } + } + ], + "site": {} + }, + + "expectedMakeRequestsErrors": [ + { + "value": "One or more invalid or missing video field(s) w, h, protocols, mimes, playbackmethod", + "comparison": "literal" + }, + { + "value": "At least one of [banner, video] formats must be defined in Imp. None found", + "comparison": "literal" + } + ] +} diff --git a/adapters/33across/usersync_test.go b/adapters/33across/usersync_test.go index fd2ebcd195b..89cae0f3f19 100644 --- a/adapters/33across/usersync_test.go +++ b/adapters/33across/usersync_test.go @@ -23,7 +23,7 @@ func Test33AcrossSyncer(t *testing.T) { Consent: "B", }, CCPA: ccpa.Policy{ - Value: "C", + Consent: "C", }, }) diff --git a/adapters/acuityads/acuityads.go b/adapters/acuityads/acuityads.go new file mode 100644 index 00000000000..b123f82cbb0 --- /dev/null +++ b/adapters/acuityads/acuityads.go @@ -0,0 +1,190 @@ +package acuityads + +import ( + "encoding/json" + "fmt" + "net/http" + "text/template" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/macros" + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/golang/glog" +) + +type AcuityAdsAdapter struct { + endpoint template.Template +} + +func NewAcuityAdsBidder(endpointTemplate string) *AcuityAdsAdapter { + template, err := template.New("endpointTemplate").Parse(endpointTemplate) + if err != nil { + glog.Fatal("Unable to parse endpoint url template") + return nil + } + return &AcuityAdsAdapter{endpoint: *template} +} + +func getHeaders(request *openrtb.BidRequest) http.Header { + headers := http.Header{} + headers.Add("Content-Type", "application/json;charset=utf-8") + headers.Add("Accept", "application/json") + headers.Add("X-Openrtb-Version", "2.5") + + if request.Device != nil { + if len(request.Device.UA) > 0 { + headers.Add("User-Agent", request.Device.UA) + } + + if len(request.Device.IPv6) > 0 { + headers.Add("X-Forwarded-For", request.Device.IPv6) + } + + if len(request.Device.IP) > 0 { + headers.Add("X-Forwarded-For", request.Device.IP) + } + } + + return headers +} + +func (a *AcuityAdsAdapter) MakeRequests( + openRTBRequest *openrtb.BidRequest, + reqInfo *adapters.ExtraRequestInfo, +) ( + requestsToBidder []*adapters.RequestData, + errs []error, +) { + + var errors []error + + acuityAdsExt, err := a.getImpressionExt(&openRTBRequest.Imp[0]) + if err != nil { + return nil, append(errors, err) + } + + url, err := a.buildEndpointURL(acuityAdsExt) + if err != nil { + return nil, []error{err} + } + + reqJSON, err := json.Marshal(openRTBRequest) + if err != nil { + return nil, []error{err} + } + + return []*adapters.RequestData{{ + Method: http.MethodPost, + Body: reqJSON, + Uri: url, + Headers: getHeaders(openRTBRequest), + }}, nil +} + +func (a *AcuityAdsAdapter) getImpressionExt(imp *openrtb.Imp) (*openrtb_ext.ExtAcuityAds, error) { + var bidderExt adapters.ExtImpBidder + if err := json.Unmarshal(imp.Ext, &bidderExt); err != nil { + return nil, &errortypes.BadInput{ + Message: "ext.bidder not provided", + } + } + var acuityAdsExt openrtb_ext.ExtAcuityAds + if err := json.Unmarshal(bidderExt.Bidder, &acuityAdsExt); err != nil { + return nil, &errortypes.BadInput{ + Message: "ext.bidder not provided", + } + } + imp.Ext = nil + return &acuityAdsExt, nil +} + +func (a *AcuityAdsAdapter) buildEndpointURL(params *openrtb_ext.ExtAcuityAds) (string, error) { + endpointParams := macros.EndpointTemplateParams{Host: params.Host, AccountID: params.AccountID} + return macros.ResolveMacros(a.endpoint, endpointParams) +} + +func (a *AcuityAdsAdapter) checkResponseStatusCodes(response *adapters.ResponseData) error { + if response.StatusCode == http.StatusNoContent { + return nil + } + + if response.StatusCode == http.StatusBadRequest { + return &errortypes.BadInput{ + Message: fmt.Sprintf("Unexpected status code: [ %d ]", response.StatusCode), + } + } + + if response.StatusCode == http.StatusServiceUnavailable { + return &errortypes.BadInput{ + Message: fmt.Sprintf("Something went wrong, please contact your Account Manager. Status Code: [ %d ] ", response.StatusCode), + } + } + + if response.StatusCode != http.StatusOK { + return &errortypes.BadInput{ + Message: fmt.Sprintf("Unexpected status code: [ %d ]. Run with request.debug = 1 for more info", response.StatusCode), + } + } + + return nil +} + +func (a *AcuityAdsAdapter) MakeBids( + openRTBRequest *openrtb.BidRequest, + requestToBidder *adapters.RequestData, + bidderRawResponse *adapters.ResponseData, +) ( + bidderResponse *adapters.BidderResponse, + errs []error, +) { + if bidderRawResponse.StatusCode == http.StatusNoContent { + return nil, nil + } + + httpStatusError := a.checkResponseStatusCodes(bidderRawResponse) + if httpStatusError != nil { + return nil, []error{httpStatusError} + } + + responseBody := bidderRawResponse.Body + var bidResp openrtb.BidResponse + if err := json.Unmarshal(responseBody, &bidResp); err != nil { + return nil, []error{&errortypes.BadServerResponse{ + Message: "Bad Server Response", + }} + } + + if len(bidResp.SeatBid) == 0 { + return nil, []error{&errortypes.BadServerResponse{ + Message: "Empty SeatBid array", + }} + } + + bidResponse := adapters.NewBidderResponseWithBidsCapacity(len(bidResp.SeatBid[0].Bid)) + sb := bidResp.SeatBid[0] + + for _, bid := range sb.Bid { + bidResponse.Bids = append(bidResponse.Bids, &adapters.TypedBid{ + Bid: &bid, + BidType: getMediaTypeForImp(bid.ImpID, openRTBRequest.Imp), + }) + } + return bidResponse, nil +} + +func getMediaTypeForImp(impId string, imps []openrtb.Imp) openrtb_ext.BidType { + mediaType := openrtb_ext.BidTypeBanner + for _, imp := range imps { + if imp.ID == impId { + if imp.Video != nil { + mediaType = openrtb_ext.BidTypeVideo + } else if imp.Native != nil { + mediaType = openrtb_ext.BidTypeNative + } + return mediaType + } + } + return mediaType +} diff --git a/adapters/acuityads/acuityads_test.go b/adapters/acuityads/acuityads_test.go new file mode 100644 index 00000000000..de44dba24ca --- /dev/null +++ b/adapters/acuityads/acuityads_test.go @@ -0,0 +1,11 @@ +package acuityads + +import ( + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adapterstest" +) + +func TestJsonSamples(t *testing.T) { + adapterstest.RunJSONBidderTest(t, "acuityadstest", NewAcuityAdsBidder("http://{{.Host}}.example.com/bid?token={{.AccountID}}")) +} diff --git a/adapters/acuityads/acuityadstest/exemplary/banner-app.json b/adapters/acuityads/acuityadstest/exemplary/banner-app.json new file mode 100644 index 00000000000..72fbfb5711e --- /dev/null +++ b/adapters/acuityads/acuityadstest/exemplary/banner-app.json @@ -0,0 +1,150 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "banner": { + "w":320, + "h":50 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "banner": { + "w":320, + "h":50 + }, + "tagid": "ogTAGID" + } + ], + "app": { + "id": "123456789", + "name": "Awesome App", + "bundle": "com.app.awesome", + "domain": "awesomeapp.com", + "cat": [ + "IAB22-1" + ], + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + { + "bid": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "1", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20", + "w":320, + "h":50 + } + ], + "type": "banner", + "seat": "acuityads" + } + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "expectedBids": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "1", + "price": 3.5, + "adm": "awesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20", + "w":320, + "h":50 + } + ] + } + \ No newline at end of file diff --git a/adapters/acuityads/acuityadstest/exemplary/banner-web.json b/adapters/acuityads/acuityadstest/exemplary/banner-web.json new file mode 100644 index 00000000000..83265367562 --- /dev/null +++ b/adapters/acuityads/acuityadstest/exemplary/banner-web.json @@ -0,0 +1,144 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "banner": { + "w":320, + "h":50 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "banner": { + "w":320, + "h":50 + } + } + ], + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + { + "bid": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "1", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20", + "w": 320, + "h": 50 + } + ], + "type": "banner", + "seat": "acuityads" + } + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "1", + "price": 3.5, + "adm": "asesome-markup", + "crid": "20", + "adomain": [ + "awesome.com" + ], + "w": 320, + "h": 50 + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/acuityads/acuityadstest/exemplary/native-app.json b/adapters/acuityads/acuityadstest/exemplary/native-app.json new file mode 100644 index 00000000000..a8fb92c942d --- /dev/null +++ b/adapters/acuityads/acuityadstest/exemplary/native-app.json @@ -0,0 +1,153 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "native": { + "ver":"1.1", + "request":"{\"adunit\":2,\"assets\":[{\"id\":3,\"img\":{\"h\":120,\"hmin\":0,\"type\":3,\"w\":180,\"wmin\":0},\"required\":1},{\"id\":0,\"required\":1,\"title\":{\"len\":25}},{\"data\":{\"len\":25,\"type\":1},\"id\":4,\"required\":1},{\"data\":{\"len\":140,\"type\":2},\"id\":6,\"required\":1}],\"context\":1,\"layout\":1,\"contextsubtype\":11,\"plcmtcnt\":1,\"plcmttype\":2,\"ver\":\"1.1\",\"ext\":{\"banner\":{\"w\":320,\"h\":50}}}" + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "native": { + "ver":"1.1", + "request":"{\"adunit\":2,\"assets\":[{\"id\":3,\"img\":{\"h\":120,\"hmin\":0,\"type\":3,\"w\":180,\"wmin\":0},\"required\":1},{\"id\":0,\"required\":1,\"title\":{\"len\":25}},{\"data\":{\"len\":25,\"type\":1},\"id\":4,\"required\":1},{\"data\":{\"len\":140,\"type\":2},\"id\":6,\"required\":1}],\"context\":1,\"layout\":1,\"contextsubtype\":11,\"plcmtcnt\":1,\"plcmttype\":2,\"ver\":\"1.1\",\"ext\":{\"banner\":{\"w\":320,\"h\":50}}}" + }, + "tagid": "ogTAGID" + } + ], + "app": { + "id": "123456789", + "name": "Awesome App", + "bundle": "com.app.awesome", + "domain": "awesomeapp.com", + "cat": [ + "IAB22-1" + ], + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + { + "bid": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "some-impression-id", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20" + } + ], + "type": "native", + "seat": "acuityads" + } + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "some-impression-id", + "price": 3.5, + "adm": "asesome-markup", + "crid": "20", + "adomain": [ + "awesome.com" + ] + }, + "type": "native" + } + ] + } + ] + } + \ No newline at end of file diff --git a/adapters/acuityads/acuityadstest/exemplary/native-web.json b/adapters/acuityads/acuityadstest/exemplary/native-web.json new file mode 100644 index 00000000000..9becd23d881 --- /dev/null +++ b/adapters/acuityads/acuityadstest/exemplary/native-web.json @@ -0,0 +1,139 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ipv6": "2607:fb90:f27:4512:d800:cb23:a603:e245", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "native": { + "ver":"1.1", + "request":"{\"adunit\":2,\"assets\":[{\"id\":3,\"img\":{\"h\":120,\"hmin\":0,\"type\":3,\"w\":180,\"wmin\":0},\"required\":1},{\"id\":0,\"required\":1,\"title\":{\"len\":25}},{\"data\":{\"len\":25,\"type\":1},\"id\":4,\"required\":1},{\"data\":{\"len\":140,\"type\":2},\"id\":6,\"required\":1}],\"context\":1,\"layout\":1,\"contextsubtype\":11,\"plcmtcnt\":1,\"plcmttype\":2,\"ver\":\"1.1\",\"ext\":{\"banner\":{\"w\":320,\"h\":50}}}" + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "2607:fb90:f27:4512:d800:cb23:a603:e245" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ipv6": "2607:fb90:f27:4512:d800:cb23:a603:e245", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "native": { + "ver":"1.1", + "request":"{\"adunit\":2,\"assets\":[{\"id\":3,\"img\":{\"h\":120,\"hmin\":0,\"type\":3,\"w\":180,\"wmin\":0},\"required\":1},{\"id\":0,\"required\":1,\"title\":{\"len\":25}},{\"data\":{\"len\":25,\"type\":1},\"id\":4,\"required\":1},{\"data\":{\"len\":140,\"type\":2},\"id\":6,\"required\":1}],\"context\":1,\"layout\":1,\"contextsubtype\":11,\"plcmtcnt\":1,\"plcmttype\":2,\"ver\":\"1.1\",\"ext\":{\"banner\":{\"w\":320,\"h\":50}}}" + } + } + ], + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + { + "bid": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "some-impression-id", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20" + } + ], + "seat": "acuityads" + } + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "some-impression-id", + "price": 3.5, + "adm": "asesome-markup", + "crid": "20", + "adomain": [ + "awesome.com" + ] + }, + "type": "native" + } + ] + } + ] +} diff --git a/adapters/acuityads/acuityadstest/exemplary/video-app.json b/adapters/acuityads/acuityadstest/exemplary/video-app.json new file mode 100644 index 00000000000..c6c75d903aa --- /dev/null +++ b/adapters/acuityads/acuityadstest/exemplary/video-app.json @@ -0,0 +1,165 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + }, + "tagid": "ogTAGID" + } + ], + "app": { + "id": "123456789", + "name": "Awesome App", + "bundle": "com.app.awesome", + "domain": "awesomeapp.com", + "cat": [ + "IAB22-1" + ], + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + { + "bid": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "some-impression-id", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20", + "w": 1280, + "h": 720 + } + ], + "seat": "acuityads" + } + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "some-impression-id", + "price": 3.5, + "adm": "asesome-markup", + "crid": "20", + "adomain": [ + "awesome.com" + ], + "w": 1280, + "h": 720 + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/acuityads/acuityadstest/exemplary/video-web.json b/adapters/acuityads/acuityadstest/exemplary/video-web.json new file mode 100644 index 00000000000..e2b9d9eb9d6 --- /dev/null +++ b/adapters/acuityads/acuityadstest/exemplary/video-web.json @@ -0,0 +1,156 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + } + } + ], + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + { + "bid": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "1", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20", + "w": 1280, + "h": 720, + "ext": { + "prebid": { + "type": "video" + } + } + } + ], + "seat": "acuityads" + } + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "expectedBids": [ + { + "id": "a3ae1b4e2fc24a4fb45540082e98e161", + "impid": "1", + "price": 3.5, + "adm": "asesome-markup", + "adomain": [ + "awesome.com" + ], + "crid": "20", + "w": 1280, + "h": 720, + "ext": { + "prebid": { + "type": "video" + } + } + } + ] +} diff --git a/adapters/acuityads/acuityadstest/params/race/banner.json b/adapters/acuityads/acuityadstest/params/race/banner.json new file mode 100644 index 00000000000..89f008afe0f --- /dev/null +++ b/adapters/acuityads/acuityadstest/params/race/banner.json @@ -0,0 +1,4 @@ +{ + "host": "ep1", + "accountid": "hash" +} diff --git a/adapters/acuityads/acuityadstest/params/race/native.json b/adapters/acuityads/acuityadstest/params/race/native.json new file mode 100644 index 00000000000..a7354b3b42a --- /dev/null +++ b/adapters/acuityads/acuityadstest/params/race/native.json @@ -0,0 +1,4 @@ +{ + "host": "ep1", + "accountid": "hash" +} \ No newline at end of file diff --git a/adapters/acuityads/acuityadstest/params/race/video.json b/adapters/acuityads/acuityadstest/params/race/video.json new file mode 100644 index 00000000000..a7354b3b42a --- /dev/null +++ b/adapters/acuityads/acuityadstest/params/race/video.json @@ -0,0 +1,4 @@ +{ + "host": "ep1", + "accountid": "hash" +} \ No newline at end of file diff --git a/adapters/acuityads/acuityadstest/supplemental/empty-seatbid-array.json b/adapters/acuityads/acuityadstest/supplemental/empty-seatbid-array.json new file mode 100644 index 00000000000..b822421ad4f --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/empty-seatbid-array.json @@ -0,0 +1,137 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + }, + "tagid": "ogTAGID" + } + ], + "app": { + "id": "123456789", + "name": "Awesome App", + "bundle": "com.app.awesome", + "domain": "awesomeapp.com", + "cat": [ + "IAB22-1" + ], + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "awesome-resp-id", + "seatbid": [ + ], + "cur": "USD", + "ext": { + "responsetimemillis": { + "acuityads": 154 + }, + "tmaxrequest": 1000 + } + } + } + } + ], + "mockResponse": { + "status": 200, + "body": "invalid response" + }, + "expectedMakeBidsErrors": [ + { + "value": "Empty SeatBid array", + "comparison": "literal" + } + ] +} diff --git a/adapters/acuityads/acuityadstest/supplemental/invalid-response.json b/adapters/acuityads/acuityadstest/supplemental/invalid-response.json new file mode 100644 index 00000000000..16ba7ada294 --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/invalid-response.json @@ -0,0 +1,112 @@ + +{ + "mockBidRequest": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "banner": { + "w":320, + "h":50 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + + "httpCalls": [{ + "expectedRequest": { + "headers": { + "Content-Type": [ + "application/json;charset=utf-8" + ], + "Accept": [ + "application/json" + ], + "X-Openrtb-Version": [ + "2.5" + ], + "User-Agent": [ + "test-user-agent" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ] + }, + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "device": { + "ua": "test-user-agent", + "ip": "123.123.123.123", + "language": "en", + "dnt": 0 + }, + "imp": [ + { + "id": "some-impression-id", + "banner": { + "w":320, + "h":50 + }, + "tagid": "ogTAGID" + } + ], + "app": { + "id": "123456789", + "name": "Awesome App", + "bundle": "com.app.awesome", + "domain": "awesomeapp.com", + "cat": [ + "IAB22-1" + ], + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 200, + "body": "invalid response" + } + }], + "expectedMakeBidsErrors": [ + { + "value": "Bad Server Response", + "comparison": "literal" + } + ] +} diff --git a/adapters/acuityads/acuityadstest/supplemental/invalid-smartyads-ext-object.json b/adapters/acuityads/acuityadstest/supplemental/invalid-smartyads-ext-object.json new file mode 100644 index 00000000000..77752d01edf --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/invalid-smartyads-ext-object.json @@ -0,0 +1,29 @@ +{ + "expectedMakeRequestsErrors": [ + { + "value": "ext.bidder not provided", + "comparison": "literal" + } + ], + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "some-impression-id", + "tagid": "my-adcode", + "video": { + "mimes": ["video/mp4"], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": "Awesome" + } + ], + "site": { + "page": "test.com" + } + }, + "httpCalls": [] +} diff --git a/adapters/acuityads/acuityadstest/supplemental/status-code-bad-request.json b/adapters/acuityads/acuityadstest/supplemental/status-code-bad-request.json new file mode 100644 index 00000000000..87b72b07f68 --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/status-code-bad-request.json @@ -0,0 +1,93 @@ + +{ + "mockBidRequest": { + "id": "some-request-id", + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + + "httpCalls": [{ + "expectedRequest": { + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "imp": [ + { + "id": "some-impression-id", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + }, + "tagid": "ogTAGID" + } + ], + "app": { + "publisher": { + "id": "123456789" + }, + "cat": [ + "IAB22-1" + ], + "bundle": "com.app.awesome", + "name": "Awesome App", + "domain": "awesomeapp.com", + "id": "123456789" + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 400 + } + }], + "expectedBidResponses": [], + "expectedMakeBidsErrors": [ + { + "value": "Unexpected status code: [ 400 ]", + "comparison": "literal" + } + ] +} diff --git a/adapters/acuityads/acuityadstest/supplemental/status-code-no-content.json b/adapters/acuityads/acuityadstest/supplemental/status-code-no-content.json new file mode 100644 index 00000000000..130710db361 --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/status-code-no-content.json @@ -0,0 +1,69 @@ +{ + "mockBidRequest": { + "id": "some-request-id", + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "imp": [{ + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": ["video/mp4"], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + }] + }, + + "httpCalls": [{ + "expectedRequest": { + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "imp": [{ + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + } + }], + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 204 + } + }], + "expectedBidResponses": [], + "expectedMakeBidsErrors": [] +} \ No newline at end of file diff --git a/adapters/acuityads/acuityadstest/supplemental/status-code-other-error.json b/adapters/acuityads/acuityadstest/supplemental/status-code-other-error.json new file mode 100644 index 00000000000..52042483b2c --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/status-code-other-error.json @@ -0,0 +1,79 @@ + +{ + "mockBidRequest": { + "id": "some-request-id", + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": ["video/mp4"], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + + "httpCalls": [{ + "expectedRequest": { + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + } + } + ], + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 306 + } + }], + "expectedBidResponses": [], + "expectedMakeBidsErrors": [ + { + "value": "Unexpected status code: [ 306 ]. Run with request.debug = 1 for more info", + "comparison": "literal" + } + ] +} diff --git a/adapters/acuityads/acuityadstest/supplemental/status-code-service-unavailable.json b/adapters/acuityads/acuityadstest/supplemental/status-code-service-unavailable.json new file mode 100644 index 00000000000..634b07cab33 --- /dev/null +++ b/adapters/acuityads/acuityadstest/supplemental/status-code-service-unavailable.json @@ -0,0 +1,79 @@ + +{ + "mockBidRequest": { + "id": "some-request-id", + "tmax": 1000, + "user": { + "buyeruid": "awesome-user" + }, + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": ["video/mp4"], + "w": 640, + "h": 480, + "minduration": 120, + "maxduration": 150 + }, + "ext": { + "bidder": { + "host": "ep1", + "accountid": "hash" + } + } + } + ] + }, + + "httpCalls": [{ + "expectedRequest": { + "uri": "http://ep1.example.com/bid?token=hash", + "body": { + "id": "some-request-id", + "imp": [ + { + "id": "some-impression-id", + "tagid": "ogTAGID", + "video": { + "mimes": [ + "video/mp4" + ], + "minduration": 120, + "maxduration": 150, + "w": 640, + "h": 480 + } + } + ], + "site": { + "page": "test.com", + "publisher": { + "id": "123456789" + } + }, + "user": { + "buyeruid": "awesome-user" + }, + "tmax": 1000 + } + }, + "mockResponse": { + "status": 503 + } + }], + "expectedBidResponses": [], + "expectedMakeBidsErrors": [ + { + "value": "Something went wrong, please contact your Account Manager. Status Code: [ 503 ] ", + "comparison": "literal" + } + ] +} diff --git a/adapters/acuityads/params_test.go b/adapters/acuityads/params_test.go new file mode 100644 index 00000000000..e1a47669796 --- /dev/null +++ b/adapters/acuityads/params_test.go @@ -0,0 +1,51 @@ +package acuityads + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" +) + +var validParams = []string{ + `{ "host": "ep1", "accountid": "hash" }`, +} + +func TestValidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + if err != nil { + t.Fatalf("Failed to fetch the json-schemas. %v", err) + } + + for _, validParam := range validParams { + if err := validator.Validate(openrtb_ext.BidderAcuityAds, json.RawMessage(validParam)); err != nil { + t.Errorf("Schema rejected AcuityAds params: %s", validParam) + } + } +} + +var invalidParams = []string{ + ``, + `null`, + `true`, + `5`, + `4.2`, + `[]`, + `{}`, + `{"adCode": "string", "seatCode": 5, "originalPublisherid": "string"}`, + `{ "accountid": "hash" }`, + `{ "host": "", "accountid": "" }`, +} + +func TestInvalidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + if err != nil { + t.Fatalf("Failed to fetch the json-schemas. %v", err) + } + + for _, invalidParam := range invalidParams { + if err := validator.Validate(openrtb_ext.BidderAcuityAds, json.RawMessage(invalidParam)); err == nil { + t.Errorf("Schema allowed unexpected params: %s", invalidParam) + } + } +} diff --git a/adapters/acuityads/usersync.go b/adapters/acuityads/usersync.go new file mode 100644 index 00000000000..90b610bb3cd --- /dev/null +++ b/adapters/acuityads/usersync.go @@ -0,0 +1,12 @@ +package acuityads + +import ( + "text/template" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/usersync" +) + +func NewAcuityAdsSyncer(temp *template.Template) usersync.Usersyncer { + return adapters.NewSyncer("acuityads", 231, temp, adapters.SyncTypeRedirect) +} diff --git a/adapters/acuityads/usersync_test.go b/adapters/acuityads/usersync_test.go new file mode 100644 index 00000000000..5c6f6a43677 --- /dev/null +++ b/adapters/acuityads/usersync_test.go @@ -0,0 +1,34 @@ +package acuityads + +import ( + "testing" + "text/template" + + "github.com/PubMatic-OpenWrap/prebid-server/privacy" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" + "github.com/stretchr/testify/assert" +) + +func TestAcuityAdsSyncer(t *testing.T) { + syncURL := "https://cs.admanmedia.com/sync/prebid?gdpr={{.GDPR}}&gdpr_consent={{.GDPRConsent}}&us_privacy={{.USPrivacy}}&redir=http%3A%2F%2Flocalhost%3A8000%2Fsetuid%3Fbidder%3Dacuityads%26uid%3D%5BUID%5D" + syncURLTemplate := template.Must( + template.New("sync-template").Parse(syncURL), + ) + syncer := NewAcuityAdsSyncer(syncURLTemplate) + syncInfo, err := syncer.GetUsersyncInfo(privacy.Policies{ + GDPR: gdpr.Policy{ + Signal: "0", + Consent: "ANDFJDS", + }, + CCPA: ccpa.Policy{ + Consent: "1-YY", + }, + }) + + assert.NoError(t, err) + assert.Equal(t, "https://cs.admanmedia.com/sync/prebid?gdpr=0&gdpr_consent=ANDFJDS&us_privacy=1-YY&redir=http%3A%2F%2Flocalhost%3A8000%2Fsetuid%3Fbidder%3Dacuityads%26uid%3D%5BUID%5D", syncInfo.URL) + assert.Equal(t, "redirect", syncInfo.Type) + assert.EqualValues(t, 231, syncer.GDPRVendorID()) + assert.Equal(t, false, syncInfo.SupportCORS) +} diff --git a/adapters/adapterstest/test_json.go b/adapters/adapterstest/test_json.go index 7bb06d0716f..b8c33064fc4 100644 --- a/adapters/adapterstest/test_json.go +++ b/adapters/adapterstest/test_json.go @@ -164,14 +164,16 @@ type httpRequest struct { } type httpResponse struct { - Status int `json:"status"` - Body json.RawMessage `json:"body"` + Status int `json:"status"` + Body json.RawMessage `json:"body"` + Headers http.Header `json:"headers"` } func (resp *httpResponse) ToResponseData(t *testing.T) *adapters.ResponseData { return &adapters.ResponseData{ StatusCode: resp.Status, Body: resp.Body, + Headers: resp.Headers, } } diff --git a/adapters/adform/adform.go b/adapters/adform/adform.go index 8b507817675..5cdef257f1c 100644 --- a/adapters/adform/adform.go +++ b/adapters/adform/adform.go @@ -42,6 +42,8 @@ type adformRequest struct { consent string digitrust *adformDigitrust currency string + eids string + url string } type adformDigitrust struct { @@ -60,6 +62,9 @@ type adformAdUnit struct { PriceType string `json:"priceType,omitempty"` KeyValues string `json:"mkv,omitempty"` KeyWords string `json:"mkw,omitempty"` + CDims string `json:"cdims,omitempty"` + MinPrice float64 `json:"minp,omitempty"` + Url string `json:"url,omitempty"` bidId string adUnitCode string @@ -74,6 +79,7 @@ type adformBid struct { Height uint64 `json:"height,omitempty"` DealId string `json:"deal_id,omitempty"` CreativeId string `json:"win_crid,omitempty"` + VastContent string `json:"vast_content,omitempty"` } const priceTypeGross = "gross" @@ -236,7 +242,8 @@ func toPBSBidSlice(adformBids []*adformBid, r *adformRequest) pbs.PBSBidSlice { bids := make(pbs.PBSBidSlice, 0) for i, bid := range adformBids { - if bid.Banner == "" || bid.ResponseType != "banner" { + adm, bidType := getAdAndType(bid) + if adm == "" { continue } pbsBid := pbs.PBSBid{ @@ -244,12 +251,12 @@ func toPBSBidSlice(adformBids []*adformBid, r *adformRequest) pbs.PBSBidSlice { AdUnitCode: r.adUnits[i].adUnitCode, BidderCode: r.bidderCode, Price: bid.Price, - Adm: bid.Banner, + Adm: adm, Width: bid.Width, Height: bid.Height, DealId: bid.DealId, Creative_id: bid.CreativeId, - CreativeMediaType: string(openrtb_ext.BidTypeBanner), + CreativeMediaType: string(bidType), } bids = append(bids, &pbsBid) @@ -279,6 +286,13 @@ func (r *adformRequest) buildAdformUrl(a *AdformAdapter) string { parameters.Add("gdpr", r.gdprApplies) parameters.Add("gdpr_consent", r.consent) + if r.eids != "" { + parameters.Add("eids", r.eids) + } + + if r.url != "" { + parameters.Add("url", r.url) + } URL := *a.URL URL.RawQuery = parameters.Encode() @@ -298,6 +312,12 @@ func (r *adformRequest) buildAdformUrl(a *AdformAdapter) string { if adUnit.KeyWords != "" { buffer.WriteString(fmt.Sprintf("&mkw=%s", adUnit.KeyWords)) } + if adUnit.CDims != "" { + buffer.WriteString(fmt.Sprintf("&cdims=%s", adUnit.CDims)) + } + if adUnit.MinPrice > 0 { + buffer.WriteString(fmt.Sprintf("&minp=%.2f", adUnit.MinPrice)) + } adUnitsParams = append(adUnitsParams, base64.URLEncoding.WithPadding(base64.NoPadding).EncodeToString(buffer.Bytes())) } @@ -403,6 +423,8 @@ func openRtbToAdformRequest(request *openrtb.BidRequest) (*adformRequest, []erro adUnits := make([]*adformAdUnit, 0, len(request.Imp)) errors := make([]error, 0, len(request.Imp)) secure := false + url := "" + for _, imp := range request.Imp { params, _, _, err := jsonparser.Get(imp.Ext, "bidder") if err != nil { @@ -437,6 +459,10 @@ func openRtbToAdformRequest(request *openrtb.BidRequest) (*adformRequest, []erro secure = true } + if url == "" { + url = adformAdUnit.Url + } + adformAdUnit.bidId = imp.ID adformAdUnit.adUnitCode = imp.ID adUnits = append(adUnits, &adformAdUnit) @@ -465,6 +491,7 @@ func openRtbToAdformRequest(request *openrtb.BidRequest) (*adformRequest, []erro } } + eids := "" consent := "" var digitrustData *openrtb_ext.ExtUserDigiTrust if request.User != nil { @@ -472,6 +499,7 @@ func openRtbToAdformRequest(request *openrtb.BidRequest) (*adformRequest, []erro if err := json.Unmarshal(request.User.Ext, &extUser); err == nil { consent = extUser.Consent digitrustData = extUser.DigiTrust + eids = encodeEids(extUser.Eids) } } @@ -513,9 +541,35 @@ func openRtbToAdformRequest(request *openrtb.BidRequest) (*adformRequest, []erro consent: consent, digitrust: digitrust, currency: requestCurrency, + eids: eids, + url: url, }, errors } +func encodeEids(eids []openrtb_ext.ExtUserEid) string { + if eids == nil { + return "" + } + + eidsMap := make(map[string]map[string][]int) + for _, eid := range eids { + _, ok := eidsMap[eid.Source] + if !ok { + eidsMap[eid.Source] = make(map[string][]int) + } + for _, uid := range eid.Uids { + eidsMap[eid.Source][uid.ID] = append(eidsMap[eid.Source][uid.ID], uid.Atype) + } + } + + encodedEids := "" + if eidsString, err := json.Marshal(eidsMap); err == nil { + encodedEids = base64.URLEncoding.WithPadding(base64.NoPadding).EncodeToString(eidsString) + } + + return encodedEids +} + func getIPSafely(device *openrtb.Device) string { if device == nil { return "" @@ -580,21 +634,23 @@ func toOpenRtbBidResponse(adformBids []*adformBid, r *openrtb.BidRequest) *adapt } for i, bid := range adformBids { - if bid.Banner == "" || bid.ResponseType != "banner" { + adm, bidType := getAdAndType(bid) + if adm == "" { continue } + openRtbBid := openrtb.Bid{ ID: r.Imp[i].ID, ImpID: r.Imp[i].ID, Price: bid.Price, - AdM: bid.Banner, + AdM: adm, W: bid.Width, H: bid.Height, DealID: bid.DealId, CrID: bid.CreativeId, } - bidResponse.Bids = append(bidResponse.Bids, &adapters.TypedBid{Bid: &openRtbBid, BidType: openrtb_ext.BidTypeBanner}) + bidResponse.Bids = append(bidResponse.Bids, &adapters.TypedBid{Bid: &openRtbBid, BidType: bidType}) currency = bid.Currency } @@ -602,3 +658,12 @@ func toOpenRtbBidResponse(adformBids []*adformBid, r *openrtb.BidRequest) *adapt return bidResponse } + +func getAdAndType(bid *adformBid) (string, openrtb_ext.BidType) { + if bid.ResponseType == "banner" { + return bid.Banner, openrtb_ext.BidTypeBanner + } else if bid.ResponseType == "vast_content" { + return bid.VastContent, openrtb_ext.BidTypeVideo + } + return "", "" +} diff --git a/adapters/adform/adform_test.go b/adapters/adform/adform_test.go index 1df0a57cb6b..76381966277 100644 --- a/adapters/adform/adform_test.go +++ b/adapters/adform/adform_test.go @@ -35,6 +35,9 @@ type aTagInfo struct { keyValues string keyWords string code string + cdims string + url string + minp float64 price float64 content string @@ -104,6 +107,16 @@ func createAdformServerResponse(testData aBidInfo) ([]byte, error) { DealId: testData.tags[2].dealId, CreativeId: testData.tags[2].creativeId, }, + { + ResponseType: "vast_content", + VastContent: testData.tags[3].content, + Price: testData.tags[3].price, + Currency: "EUR", + Width: testData.width, + Height: testData.height, + DealId: testData.tags[3].dealId, + CreativeId: testData.tags[3].creativeId, + }, } adformServerResponse, err := json.Marshal(bids) return adformServerResponse, err @@ -120,10 +133,21 @@ func TestAdformBasicResponse(t *testing.T) { if err != nil { t.Fatalf("Should not have gotten adapter error: %v", err) } - if len(bids) != 2 { - t.Fatalf("Received %d bids instead of 2", len(bids)) + if len(bids) != 3 { + t.Fatalf("Received %d bids instead of 3", len(bids)) + } + expectedTypes := []openrtb_ext.BidType{ + openrtb_ext.BidTypeBanner, + openrtb_ext.BidTypeBanner, + openrtb_ext.BidTypeVideo, } - for _, bid := range bids { + + for i, bid := range bids { + + if bid.CreativeMediaType != string(expectedTypes[i]) { + t.Errorf("Expected a %s bid. Got: %s", expectedTypes[i], bid.CreativeMediaType) + } + matched := false for _, tag := range adformTestData.tags { if bid.AdUnitCode == tag.code { @@ -221,7 +245,7 @@ func preparePrebidRequest(serverUrl string, t *testing.T) *pbs.PBSRequest { func preparePrebidRequestBody(requestData aBidInfo, t *testing.T) *bytes.Buffer { prebidRequest := pbs.PBSRequest{ - AdUnits: make([]pbs.AdUnit, 3), + AdUnits: make([]pbs.AdUnit, 4), Device: &openrtb.Device{ UA: requestData.deviceUA, IP: requestData.deviceIP, @@ -320,9 +344,10 @@ func createTestData(secure bool) aBidInfo { tid: "transaction-id", buyerUID: "user-id", tags: []aTagInfo{ - {mid: 32344, keyValues: "color:red,age:30-40", keyWords: "red,blue", priceType: "gross", code: "code1", price: 1.23, content: "banner-content1", dealId: "dealId1", creativeId: "creativeId1"}, - {mid: 32345, priceType: "net", code: "code2"}, // no bid for ad unit - {mid: 32346, code: "code3", price: 1.24, content: "banner-content2", dealId: "dealId2"}, + {mid: 32344, keyValues: "color:red,age:30-40", keyWords: "red,blue", cdims: "300x300,400x200", priceType: "gross", code: "code1", price: 1.23, content: "banner-content1", dealId: "dealId1", creativeId: "creativeId1"}, + {mid: 32345, priceType: "net", code: "code2", minp: 23.1, cdims: "300x200"}, // no bid for ad unit + {mid: 32346, code: "code3", price: 1.24, content: "banner-content2", dealId: "dealId2", url: "https://adform.com?a=b"}, + {mid: 32347, code: "code4", content: "vast-xml"}, }, secure: secure, currency: "EUR", @@ -376,6 +401,11 @@ func createOpenRtbRequest(testData *aBidInfo) *openrtb.BidRequest { func TestOpenRTBStandardResponse(t *testing.T) { testData := createTestData(true) request := createOpenRtbRequest(&testData) + expectedTypes := []openrtb_ext.BidType{ + openrtb_ext.BidTypeBanner, + openrtb_ext.BidTypeBanner, + openrtb_ext.BidTypeVideo, + } responseBody, err := createAdformServerResponse(testData) if err != nil { @@ -387,16 +417,17 @@ func TestOpenRTBStandardResponse(t *testing.T) { bidder := new(AdformAdapter) bidResponse, errs := bidder.MakeBids(request, nil, httpResponse) - if len(bidResponse.Bids) != 2 { - t.Fatalf("Expected 2 bids. Got %d", len(bidResponse.Bids)) + if len(bidResponse.Bids) != 3 { + t.Fatalf("Expected 3 bids. Got %d", len(bidResponse.Bids)) } if len(errs) != 0 { t.Errorf("Expected 0 errors. Got %d", len(errs)) } - for _, typeBid := range bidResponse.Bids { - if typeBid.BidType != openrtb_ext.BidTypeBanner { - t.Errorf("Expected a banner bid. Got: %s", bidResponse.Bids[0].BidType) + for i, typeBid := range bidResponse.Bids { + + if typeBid.BidType != expectedTypes[i] { + t.Errorf("Expected a %s bid. Got: %s", expectedTypes[i], typeBid.BidType) } bid := typeBid.Bid matched := false @@ -480,7 +511,33 @@ func getUserExt() []byte { KeyV: 1, Pref: 0, } + + eids := []openrtb_ext.ExtUserEid{ + { + Source: "test.com", + Uids: []openrtb_ext.ExtUserEidUid{ + { + ID: "some_user_id", + Atype: 1, + }, + { + ID: "other_user_id", + }, + }, + }, + { + Source: "test2.org", + Uids: []openrtb_ext.ExtUserEidUid{ + { + ID: "other_user_id", + Atype: 2, + }, + }, + }, + } + userExt := openrtb_ext.ExtUser{ + Eids: eids, Consent: "abc", DigiTrust: &digitrust, } @@ -493,11 +550,22 @@ func getUserExt() []byte { } func formatAdUnitJson(tag aTagInfo) string { - return fmt.Sprintf("{ \"mid\": %d%s%s%s}", + return fmt.Sprintf("{ \"mid\": %d%s%s%s%s%s%s}", tag.mid, formatAdUnitParam("priceType", tag.priceType), formatAdUnitParam("mkv", tag.keyValues), - formatAdUnitParam("mkw", tag.keyWords)) + formatAdUnitParam("mkw", tag.keyWords), + formatAdUnitParam("cdims", tag.cdims), + formatAdUnitParam("url", tag.url), + formatDemicalAdUnitParam("minp", tag.minp)) +} + +func formatDemicalAdUnitParam(fieldName string, fieldValue float64) string { + if fieldValue > 0 { + return fmt.Sprintf(", \"%s\": %.2f", fieldName, fieldValue) + } + + return "" } func formatAdUnitParam(fieldName string, fieldValue string) string { @@ -519,13 +587,16 @@ func assertAdformServerRequest(testData aBidInfo, r *http.Request, isOpenRtb boo } var midsWithCurrency = "" + var queryString = "" if isOpenRtb { - midsWithCurrency = "bWlkPTMyMzQ0JnJjdXI9RVVSJm1rdj1jb2xvcjpyZWQsYWdlOjMwLTQwJm1rdz1yZWQsYmx1ZQ&bWlkPTMyMzQ1JnJjdXI9RVVS&bWlkPTMyMzQ2JnJjdXI9RVVS" + midsWithCurrency = "bWlkPTMyMzQ0JnJjdXI9RVVSJm1rdj1jb2xvcjpyZWQsYWdlOjMwLTQwJm1rdz1yZWQsYmx1ZSZjZGltcz0zMDB4MzAwLDQwMHgyMDA&bWlkPTMyMzQ1JnJjdXI9RVVSJmNkaW1zPTMwMHgyMDAmbWlucD0yMy4xMA&bWlkPTMyMzQ2JnJjdXI9RVVS&bWlkPTMyMzQ3JnJjdXI9RVVS" + queryString = "CC=1&adid=6D92078A-8246-4BA4-AE5B-76104861E7DC&eids=eyJ0ZXN0LmNvbSI6eyJvdGhlcl91c2VyX2lkIjpbMF0sInNvbWVfdXNlcl9pZCI6WzFdfSwidGVzdDIub3JnIjp7Im90aGVyX3VzZXJfaWQiOlsyXX19&fd=1&gdpr=1&gdpr_consent=abc&ip=111.111.111.111&pt=gross&rp=4&stid=transaction-id&url=https%3A%2F%2Fadform.com%3Fa%3Db&" + midsWithCurrency } else { - midsWithCurrency = "bWlkPTMyMzQ0JnJjdXI9VVNEJm1rdj1jb2xvcjpyZWQsYWdlOjMwLTQwJm1rdz1yZWQsYmx1ZQ&bWlkPTMyMzQ1JnJjdXI9VVNE&bWlkPTMyMzQ2JnJjdXI9VVNE" // no way to pass currency in legacy adapter + midsWithCurrency = "bWlkPTMyMzQ0JnJjdXI9VVNEJm1rdj1jb2xvcjpyZWQsYWdlOjMwLTQwJm1rdz1yZWQsYmx1ZSZjZGltcz0zMDB4MzAwLDQwMHgyMDA&bWlkPTMyMzQ1JnJjdXI9VVNEJmNkaW1zPTMwMHgyMDAmbWlucD0yMy4xMA&bWlkPTMyMzQ2JnJjdXI9VVNE&bWlkPTMyMzQ3JnJjdXI9VVNE" // no way to pass currency in legacy adapter + queryString = "CC=1&adid=6D92078A-8246-4BA4-AE5B-76104861E7DC&fd=1&gdpr=1&gdpr_consent=abc&ip=111.111.111.111&pt=gross&rp=4&stid=transaction-id&" + midsWithCurrency } - if ok, err := equal("CC=1&adid=6D92078A-8246-4BA4-AE5B-76104861E7DC&fd=1&gdpr=1&gdpr_consent=abc&ip=111.111.111.111&pt=gross&rp=4&stid=transaction-id&"+midsWithCurrency, r.URL.RawQuery, "Query string"); !ok { + if ok, err := equal(queryString, r.URL.RawQuery, "Query string"); !ok { return err } if ok, err := equal("application/json;charset=utf-8", r.Header.Get("Content-Type"), "Content type"); !ok { @@ -603,7 +674,7 @@ func TestPriceTypeUrlParameterCreation(t *testing.T) { // Asserts that toOpenRtbBidResponse() creates a *adapters.BidderResponse with // the currency of the last valid []*adformBid element and the expected number of bids func TestToOpenRtbBidResponse(t *testing.T) { - expectedBids := 3 + expectedBids := 4 lastCurrency, anotherCurrency, emptyCurrency := "EUR", "USD", "" request := &openrtb.BidRequest{ @@ -629,6 +700,11 @@ func TestToOpenRtbBidResponse(t *testing.T) { Ext: json.RawMessage(`{"bidder1": { "mid": "32344" }}`), Banner: &openrtb.Banner{}, }, + { + ID: "video-imp-no4", + Ext: json.RawMessage(`{"bidder1": { "mid": "32345" }}`), + Banner: &openrtb.Banner{}, + }, }, Device: &openrtb.Device{UA: "ua", IP: "ip"}, User: &openrtb.User{BuyerUID: "buyerUID"}, @@ -660,6 +736,16 @@ func TestToOpenRtbBidResponse(t *testing.T) { ResponseType: "banner", Banner: "banner-content4", Price: 1.25, + Currency: emptyCurrency, + Width: 300, + Height: 200, + DealId: "dealId4", + CreativeId: "creativeId4", + }, + { + ResponseType: "vast_content", + VastContent: "vast-content", + Price: 1.25, Currency: lastCurrency, Width: 300, Height: 200, diff --git a/adapters/adform/adformtest/exemplary/multiformat-impression.json b/adapters/adform/adformtest/exemplary/multiformat-impression.json new file mode 100644 index 00000000000..efd4aca63e2 --- /dev/null +++ b/adapters/adform/adformtest/exemplary/multiformat-impression.json @@ -0,0 +1,99 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "banner-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + } + ] + }, + "ext": { + "bidder": { + "mid": 12345 + } + } + }, + { + "id": "video-imp-id", + "video": { + "w": 640, + "h": 480 + }, + "ext": { + "bidder": { + "mid": 54321 + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://adx.adform.net/adx?CC=1&fd=1&gdpr=&gdpr_consent=&ip=&rp=4&stid=&bWlkPTEyMzQ1JnJjdXI9VVNE&bWlkPTU0MzIxJnJjdXI9VVNE" + }, + "mockResponse": { + "status": 200, + "body": [ + { + "response": "banner", + "banner": "", + "win_bid": 0.5, + "win_cur": "USD", + "width": 300, + "height": 250, + "deal_id": null, + "win_crid": "20078830" + }, + { + "response": "vast_content", + "vast_content": "", + "win_bid": 0.7, + "win_cur": "USD", + "width": 640, + "height": 480, + "deal_id": "DID-123-22", + "win_crid": "20078831" + } + ] + } + } + ], + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "banner-imp-id", + "impid": "banner-imp-id", + "price": 0.5, + "adm": "", + "crid": "20078830", + "w": 300, + "h": 250 + }, + "type": "banner" + }, + { + "bid": { + "id": "video-imp-id", + "impid": "video-imp-id", + "price": 0.7, + "adm": "", + "crid": "20078831", + "dealid": "DID-123-22", + "w": 640, + "h": 480 + }, + "type": "video" + } + ] + } + ] + } diff --git a/adapters/adform/adformtest/exemplary/single-banner-impression.json b/adapters/adform/adformtest/exemplary/single-banner-impression.json new file mode 100644 index 00000000000..fd7f3cde526 --- /dev/null +++ b/adapters/adform/adformtest/exemplary/single-banner-impression.json @@ -0,0 +1,64 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + } + ] + }, + "ext": { + "bidder": { + "mid": 12345 + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://adx.adform.net/adx?CC=1&fd=1&gdpr=&gdpr_consent=&ip=&rp=4&stid=&bWlkPTEyMzQ1JnJjdXI9VVNE" + }, + "mockResponse": { + "status": 200, + "body": [ + { + "response": "banner", + "banner": "", + "win_bid": 0.5, + "win_cur": "USD", + "width": 300, + "height": 250, + "deal_id": null, + "win_crid": "20078830" + } + ] + } + } + ], + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "test-imp-id", + "impid": "test-imp-id", + "price": 0.5, + "adm": "", + "crid": "20078830", + "w": 300, + "h": 250 + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/adform/adformtest/exemplary/single-video-impression.json b/adapters/adform/adformtest/exemplary/single-video-impression.json new file mode 100644 index 00000000000..e22977e6523 --- /dev/null +++ b/adapters/adform/adformtest/exemplary/single-video-impression.json @@ -0,0 +1,60 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "video": { + "w": 640, + "h": 480 + }, + "ext": { + "bidder": { + "mid": 54321 + } + } + } + ] + }, + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://adx.adform.net/adx?CC=1&fd=1&gdpr=&gdpr_consent=&ip=&rp=4&stid=&bWlkPTU0MzIxJnJjdXI9VVNE" + }, + "mockResponse": { + "status": 200, + "body": [ + { + "response": "vast_content", + "vast_content": "", + "win_bid": 0.5, + "win_cur": "USD", + "width": 640, + "height": 480, + "deal_id": null, + "win_crid": "20078830" + } + ] + } + } + ], + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "test-imp-id", + "impid": "test-imp-id", + "price": 0.5, + "adm": "", + "crid": "20078830", + "w": 640, + "h": 480 + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/adform/adformtest/params/race/video.json b/adapters/adform/adformtest/params/race/video.json new file mode 100644 index 00000000000..51f8f1b94d2 --- /dev/null +++ b/adapters/adform/adformtest/params/race/video.json @@ -0,0 +1,3 @@ +{ + "mid": "858300" +} diff --git a/adapters/adform/params_test.go b/adapters/adform/params_test.go index 98596075c74..c4ffb79399e 100644 --- a/adapters/adform/params_test.go +++ b/adapters/adform/params_test.go @@ -48,6 +48,10 @@ var validParams = []string{ `{"mid":"123","mkv":"color:"}`, `{"mid":"123","mkw":"green,male"}`, `{"mid":"123","mkv":" ","mkw":" "}`, + `{"mid":"123","cdims":"500x300,400x200","mkw":" "}`, + `{"mid":"123","cdims":"500x300","mkv":" ","mkw":" "}`, + `{"mid":"123","minp":2.1}`, + `{"mid":"123","url":"https://adform.com/page"}`, } var invalidParams = []string{ @@ -66,4 +70,8 @@ var invalidParams = []string{ `{"mid":"123","mkv":"color:blue,l&ngth:350"}`, `{"mid":"123","mkv":"color::blue"}`, `{"mid":"123","mkw":"fem&le"}`, + `{"mid":"123","minp":"2.1"}`, + `{"mid":"123","cdims":"500x300:400:200","mkw":" "}`, + `{"mid":"123","cdims":"500x300,400:200","mkv":" ","mkw":" "}`, + `{"mid":"123","url":10}`, } diff --git a/adapters/adgeneration/adgeneration.go b/adapters/adgeneration/adgeneration.go index 069609f4262..376933734bd 100644 --- a/adapters/adgeneration/adgeneration.go +++ b/adapters/adgeneration/adgeneration.go @@ -54,6 +54,9 @@ func (adg *AdgenerationAdapter) MakeRequests(request *openrtb.BidRequest, reqInf headers := http.Header{} headers.Add("Content-Type", "application/json;charset=utf-8") headers.Add("Accept", "application/json") + if request.Device != nil && len(request.Device.UA) > 0 { + headers.Add("User-Agent", request.Device.UA) + } bidRequestArray := make([]*adapters.RequestData, 0, numRequests) @@ -106,11 +109,14 @@ func (adg *AdgenerationAdapter) getRawQuery(id string, request *openrtb.BidReque v.Set("adapterver", adg.version) adSize := getSizes(imp) if adSize != "" { - v.Set("size", adSize) + v.Set("sizes", adSize) } if request.Site != nil && request.Site.Page != "" { v.Set("tp", request.Site.Page) } + if request.Source != nil && request.Source.TID != "" { + v.Set("transactionid", request.Source.TID) + } return &v } @@ -135,7 +141,7 @@ func getSizes(imp *openrtb.Imp) string { } var sizeStr string for _, v := range imp.Banner.Format { - sizeStr += strconv.FormatUint(v.W, 10) + "×" + strconv.FormatUint(v.H, 10) + "," + sizeStr += strconv.FormatUint(v.W, 10) + "x" + strconv.FormatUint(v.H, 10) + "," } if len(sizeStr) > 0 && strings.LastIndex(sizeStr, ",") == len(sizeStr)-1 { sizeStr = sizeStr[:len(sizeStr)-1] @@ -210,6 +216,7 @@ func (adg *AdgenerationAdapter) MakeBids(internalRequest *openrtb.BidRequest, ex Bid: &bid, BidType: bitType, }) + bidResponse.Currency = adg.getCurrency(internalRequest) return bidResponse, nil } } @@ -254,7 +261,7 @@ func removeWrapper(ad string) string { func NewAdgenerationAdapter(endpoint string) *AdgenerationAdapter { return &AdgenerationAdapter{ endpoint, - "1.0.0", + "1.0.2", "JPY", } } diff --git a/adapters/adgeneration/adgeneration_test.go b/adapters/adgeneration/adgeneration_test.go index 2c679e10471..d6152dc760b 100644 --- a/adapters/adgeneration/adgeneration_test.go +++ b/adapters/adgeneration/adgeneration_test.go @@ -5,14 +5,16 @@ import ( "testing" "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adapterstest" + "github.com/stretchr/testify/assert" ) func TestJsonSamples(t *testing.T) { adapterstest.RunJSONBidderTest(t, "adgenerationtest", NewAdgenerationAdapter("https://d.socdm.com/adsv/v1")) } -func TestgetRequestUri(t *testing.T) { +func TestGetRequestUri(t *testing.T) { bidder := NewAdgenerationAdapter("https://d.socdm.com/adsv/v1") // Test items failedRequest := &openrtb.BidRequest{ @@ -22,6 +24,7 @@ func TestgetRequestUri(t *testing.T) { {ID: "extImpBidder-failed-test", Banner: &openrtb.Banner{Format: []openrtb.Format{{W: 300, H: 250}}}, Ext: json.RawMessage(`{"_bidder": { "id": "58278" }}`)}, {ID: "extImpAdgeneration-failed-test", Banner: &openrtb.Banner{Format: []openrtb.Format{{W: 300, H: 250}}}, Ext: json.RawMessage(`{"bidder": { "_id": "58278" }}`)}, }, + Source: &openrtb.Source{TID: "SourceTID"}, Device: &openrtb.Device{UA: "testUA", IP: "testIP"}, Site: &openrtb.Site{Page: "https://supership.com"}, User: &openrtb.User{BuyerUID: "buyerID"}, @@ -31,6 +34,7 @@ func TestgetRequestUri(t *testing.T) { Imp: []openrtb.Imp{ {ID: "bidRequest-success-test", Banner: &openrtb.Banner{Format: []openrtb.Format{{W: 300, H: 250}}}, Ext: json.RawMessage(`{"bidder": { "id": "58278" }}`)}, }, + Source: &openrtb.Source{TID: "SourceTID"}, Device: &openrtb.Device{UA: "testUA", IP: "testIP"}, Site: &openrtb.Site{Page: "https://supership.com"}, User: &openrtb.User{BuyerUID: "buyerID"}, @@ -48,14 +52,6 @@ func TestgetRequestUri(t *testing.T) { } numRequests = len(successRequest.Imp) for index := 0; index < numRequests; index++ { - // RequestUri Test. - httpRequests, err := bidder.getRequestUri(successRequest, index) - if err != nil { - t.Errorf("getRequestUri: %v did throw an error: %v", successRequest.Imp[index], err) - } - if httpRequests == "adapterver="+bidder.version+"¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&size=300%C3%97250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html" { - t.Errorf("getRequestUri: %v did return Request: %s", successRequest.Imp[index], httpRequests) - } // getRawQuery Test. adgExt, err := unmarshalExtImpAdgeneration(&successRequest.Imp[index]) if err != nil { @@ -63,27 +59,33 @@ func TestgetRequestUri(t *testing.T) { } rawQuery := bidder.getRawQuery(adgExt.Id, successRequest, &successRequest.Imp[index]) expectQueries := map[string]string{ - "posall": "SSPLOC", - "id": adgExt.Id, - "sdktype": "0", - "hb": "true", - "currency": bidder.getCurrency(successRequest), - "sdkname": "prebidserver", - "adapterver": bidder.version, - "size": getSizes(&successRequest.Imp[index]), - "tp": successRequest.Site.Name, + "posall": "SSPLOC", + "id": adgExt.Id, + "sdktype": "0", + "hb": "true", + "currency": bidder.getCurrency(successRequest), + "sdkname": "prebidserver", + "adapterver": bidder.version, + "sizes": getSizes(&successRequest.Imp[index]), + "tp": successRequest.Site.Page, + "transactionid": successRequest.Source.TID, } for key, expectedValue := range expectQueries { actualValue := rawQuery.Get(key) - if actualValue == "" { - if !(key == "size" || key == "tp") { - t.Errorf("getRawQuery: key %s is required value.", key) - } - } if actualValue != expectedValue { t.Errorf("getRawQuery: %s value does not match expected %s, actual %s", key, expectedValue, actualValue) } } + + // RequestUri Test. + actualUri, err := bidder.getRequestUri(successRequest, index) + if err != nil { + t.Errorf("getRequestUri: %v did throw an error: %v", successRequest.Imp[index], err) + } + expectedUri := "https://d.socdm.com/adsv/v1?adapterver=" + bidder.version + "¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&sizes=300x250&t=json3&tp=https%3A%2F%2Fsupership.com&transactionid=SourceTID" + if actualUri != expectedUri { + t.Errorf("getRequestUri: does not match expected %s, actual %s", expectedUri, actualUri) + } } } @@ -97,7 +99,7 @@ func TestGetSizes(t *testing.T) { request = &openrtb.Imp{Banner: multiFormatBanner} size = getSizes(request) - if size != "300×250,320×50" { + if size != "300x250,320x50" { t.Errorf("%v does not match size.", multiFormatBanner) } request = &openrtb.Imp{Banner: noFormatBanner} @@ -174,3 +176,64 @@ func TestCreateAd(t *testing.T) { t.Errorf("%v does not match createAd.", adgVastResponse) } } + +func TestMakeBids(t *testing.T) { + bidder := NewAdgenerationAdapter("https://d.socdm.com/adsv/v1") + internalRequest := &openrtb.BidRequest{ + ID: "test-success-bid-request", + Imp: []openrtb.Imp{ + {ID: "bidRequest-success-test", Banner: &openrtb.Banner{Format: []openrtb.Format{{W: 300, H: 250}}}, Ext: json.RawMessage(`{"bidder": { "id": "58278" }}`)}, + }, + Device: &openrtb.Device{UA: "testUA", IP: "testIP"}, + Site: &openrtb.Site{Page: "https://supership.com"}, + User: &openrtb.User{BuyerUID: "buyerID"}, + } + externalRequest := adapters.RequestData{} + response := adapters.ResponseData{ + StatusCode: 200, + Body: ([]byte)("{\n \"ad\": \"testAd\",\n \"cpm\": 30,\n \"creativeid\": \"Dummy_supership.jp\",\n \"h\": 250,\n \"locationid\": \"58278\",\n \"results\": [{}],\n \"dealid\": \"test-deal-id\",\n \"w\": 300\n }"), + } + // default Currency InternalRequest + defaultCurBidderResponse, errs := bidder.MakeBids(internalRequest, &externalRequest, &response) + if len(errs) > 0 { + t.Errorf("MakeBids return errors. errors: %v", errs) + } + checkBidResponse(t, defaultCurBidderResponse, bidder.defaultCurrency) + + // Specified Currency InternalRequest + usdCur := "USD" + internalRequest.Cur = []string{usdCur} + specifiedCurBidderResponse, errs := bidder.MakeBids(internalRequest, &externalRequest, &response) + if len(errs) > 0 { + t.Errorf("MakeBids return errors. errors: %v", errs) + } + checkBidResponse(t, specifiedCurBidderResponse, usdCur) + +} + +func checkBidResponse(t *testing.T, bidderResponse *adapters.BidderResponse, expectedCurrency string) { + if bidderResponse == nil { + t.Errorf("actual bidResponse is nil.") + } + + // AdM is assured by TestCreateAd and JSON tests + var expectedAdM string = "testAd" + var expectedID string = "58278" + var expectedImpID = "bidRequest-success-test" + var expectedPrice float64 = 30.0 + var expectedW uint64 = 300 + var expectedH uint64 = 250 + var expectedCrID string = "Dummy_supership.jp" + var extectedDealID string = "test-deal-id" + + assert.Equal(t, expectedCurrency, bidderResponse.Currency) + assert.Equal(t, 1, len(bidderResponse.Bids)) + assert.Equal(t, expectedID, bidderResponse.Bids[0].Bid.ID) + assert.Equal(t, expectedImpID, bidderResponse.Bids[0].Bid.ImpID) + assert.Equal(t, expectedAdM, bidderResponse.Bids[0].Bid.AdM) + assert.Equal(t, expectedPrice, bidderResponse.Bids[0].Bid.Price) + assert.Equal(t, expectedW, bidderResponse.Bids[0].Bid.W) + assert.Equal(t, expectedH, bidderResponse.Bids[0].Bid.H) + assert.Equal(t, expectedCrID, bidderResponse.Bids[0].Bid.CrID) + assert.Equal(t, extectedDealID, bidderResponse.Bids[0].Bid.DealID) +} diff --git a/adapters/adgeneration/adgenerationtest/exemplary/single-banner.json b/adapters/adgeneration/adgenerationtest/exemplary/single-banner.json index d23a510bee5..655f6b75f91 100644 --- a/adapters/adgeneration/adgenerationtest/exemplary/single-banner.json +++ b/adapters/adgeneration/adgenerationtest/exemplary/single-banner.json @@ -4,6 +4,9 @@ "site": { "page": "http://example.com/test.html" }, + "device": { + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" + }, "imp": [ { "id": "some-impression-id", @@ -52,13 +55,16 @@ "tmax": 500 }, "expectedRequest":{ - "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.0¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&size=300%C3%97250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", + "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.2¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&sizes=300x250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", "headers": { "Accept": [ "application/json" ], "Content-Type": [ "application/json;charset=utf-8" + ], + "User-Agent": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" ] } }, diff --git a/adapters/adgeneration/adgenerationtest/supplemental/204-bid-response.json b/adapters/adgeneration/adgenerationtest/supplemental/204-bid-response.json index cf8635bbc3d..8a4135c27af 100644 --- a/adapters/adgeneration/adgenerationtest/supplemental/204-bid-response.json +++ b/adapters/adgeneration/adgenerationtest/supplemental/204-bid-response.json @@ -4,6 +4,9 @@ "site": { "page": "http://example.com/test.html" }, + "device": { + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" + }, "imp": [ { "id": "some-impression-id", @@ -52,13 +55,16 @@ "tmax": 500 }, "expectedRequest":{ - "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.0¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&size=300%C3%97250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", + "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.2¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&sizes=300x250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", "headers": { "Accept": [ "application/json" ], "Content-Type": [ "application/json;charset=utf-8" + ], + "User-Agent": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" ] } }, @@ -69,4 +75,4 @@ } ], "expectedBidResponses": [] -} \ No newline at end of file +} diff --git a/adapters/adgeneration/adgenerationtest/supplemental/400-bid-response.json b/adapters/adgeneration/adgenerationtest/supplemental/400-bid-response.json index f5dc7fe0af5..ebda348fa65 100644 --- a/adapters/adgeneration/adgenerationtest/supplemental/400-bid-response.json +++ b/adapters/adgeneration/adgenerationtest/supplemental/400-bid-response.json @@ -4,6 +4,9 @@ "site": { "page": "http://example.com/test.html" }, + "device": { + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" + }, "imp": [ { "id": "some-impression-id", @@ -52,13 +55,16 @@ "tmax": 500 }, "expectedRequest":{ - "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.0¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&size=300%C3%97250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", + "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.2¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&sizes=300x250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", "headers": { "Accept": [ "application/json" ], "Content-Type": [ "application/json;charset=utf-8" + ], + "User-Agent": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" ] } }, @@ -74,4 +80,4 @@ "comparison": "literal" } ] -} \ No newline at end of file +} diff --git a/adapters/adgeneration/adgenerationtest/supplemental/no-bid-response.json b/adapters/adgeneration/adgenerationtest/supplemental/no-bid-response.json index 399f85a5856..3b3bc82a4ce 100644 --- a/adapters/adgeneration/adgenerationtest/supplemental/no-bid-response.json +++ b/adapters/adgeneration/adgenerationtest/supplemental/no-bid-response.json @@ -4,6 +4,9 @@ "site": { "page": "http://example.com/test.html" }, + "device": { + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" + }, "imp": [ { "id": "some-impression-id", @@ -52,13 +55,16 @@ "tmax": 500 }, "expectedRequest":{ - "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.0¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&size=300%C3%97250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", + "uri": "https://d.socdm.com/adsv/v1?adapterver=1.0.2¤cy=JPY&hb=true&id=58278&posall=SSPLOC&sdkname=prebidserver&sdktype=0&sizes=300x250&t=json3&tp=http%3A%2F%2Fexample.com%2Ftest.html", "headers": { "Accept": [ "application/json" ], "Content-Type": [ "application/json;charset=utf-8" + ], + "User-Agent": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" ] } }, diff --git a/adapters/adhese/adhese.go b/adapters/adhese/adhese.go index 1c150d27057..c6a16ed051e 100644 --- a/adapters/adhese/adhese.go +++ b/adapters/adhese/adhese.go @@ -73,6 +73,13 @@ func extractRefererParameter(request *openrtb.BidRequest) string { return "" } +func extractIfaParameter(request *openrtb.BidRequest) string { + if request.Device != nil && request.Device.IFA != "" { + return "/xz" + url.QueryEscape(request.Device.IFA) + } + return "" +} + func (a *AdheseAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *adapters.ExtraRequestInfo) ([]*adapters.RequestData, []error) { errs := make([]error, 0, len(request.Imp)) @@ -106,12 +113,13 @@ func (a *AdheseAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *adapt errs = append(errs, WrapReqError("Could not compose url from template and request account val: "+err.Error())) return nil, errs } - complete_url := fmt.Sprintf("%s%s%s%s%s", + complete_url := fmt.Sprintf("%s%s%s%s%s%s", host, extractSlotParameter(params), extractTargetParameters(params), extractGdprParameter(request), - extractRefererParameter(request)) + extractRefererParameter(request), + extractIfaParameter(request)) return []*adapters.RequestData{{ Method: "GET", diff --git a/adapters/adhese/adhesetest/exemplary/banner-internal.json b/adapters/adhese/adhesetest/exemplary/banner-internal.json index 3a31d0ccf3c..225b37aa2f8 100644 --- a/adapters/adhese/adhesetest/exemplary/banner-internal.json +++ b/adapters/adhese/adhesetest/exemplary/banner-internal.json @@ -37,12 +37,15 @@ "publisher": { "id": "123" } + }, + "device": { + "IFA": "dum-my" } }, "httpCalls": [ { "expectedRequest": { - "uri": "https://ads-demo.adhese.com/json/sl_adhese_prebid_demo_-leaderboard/ag55/cigent;brussels/tlall/xtdummy" + "uri": "https://ads-demo.adhese.com/json/sl_adhese_prebid_demo_-leaderboard/ag55/cigent;brussels/tlall/xtdummy/xzdum-my" }, "mockResponse": { "status": 200, @@ -100,4 +103,4 @@ ] } ] -} \ No newline at end of file +} diff --git a/adapters/adkernel/usersync_test.go b/adapters/adkernel/usersync_test.go index 53435d5e3a0..7230fcbab9c 100644 --- a/adapters/adkernel/usersync_test.go +++ b/adapters/adkernel/usersync_test.go @@ -23,7 +23,7 @@ func TestAdkernelAdnSyncer(t *testing.T) { Consent: "BONciguONcjGKADACHENAOLS1rAHDAFAAEAASABQAMwAeACEAFw", }, CCPA: ccpa.Policy{ - Value: "1NYN", + Consent: "1NYN", }, }) diff --git a/adapters/adkernelAdn/usersync_test.go b/adapters/adkernelAdn/usersync_test.go index 2f90e73d348..9528579aa3d 100644 --- a/adapters/adkernelAdn/usersync_test.go +++ b/adapters/adkernelAdn/usersync_test.go @@ -23,7 +23,7 @@ func TestAdkernelAdnSyncer(t *testing.T) { Consent: "BONciguONcjGKADACHENAOLS1rAHDAFAAEAASABQAMwAeACEAFw", }, CCPA: ccpa.Policy{ - Value: "1NYN", + Consent: "1NYN", }, }) diff --git a/adapters/adman/adman.go b/adapters/adman/adman.go new file mode 100644 index 00000000000..b6276a9fac3 --- /dev/null +++ b/adapters/adman/adman.go @@ -0,0 +1,140 @@ +package adman + +import ( + "encoding/json" + "fmt" + "net/http" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" +) + +// AdmanAdapter struct +type AdmanAdapter struct { + URI string +} + +// NewAdmanBidder Initializes the Bidder +func NewAdmanBidder(endpoint string) *AdmanAdapter { + return &AdmanAdapter{ + URI: endpoint, + } +} + +type admanParams struct { + TagID string `json:"TagID"` +} + +// MakeRequests create bid request for adman demand +func (a *AdmanAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *adapters.ExtraRequestInfo) ([]*adapters.RequestData, []error) { + var errs []error + var admanExt openrtb_ext.ExtImpAdman + var err error + + var adapterRequests []*adapters.RequestData + + reqCopy := *request + for _, imp := range request.Imp { + reqCopy.Imp = []openrtb.Imp{imp} + + var bidderExt adapters.ExtImpBidder + if err = json.Unmarshal(reqCopy.Imp[0].Ext, &bidderExt); err != nil { + errs = append(errs, err) + continue + } + + if err = json.Unmarshal(bidderExt.Bidder, &admanExt); err != nil { + errs = append(errs, err) + continue + } + + reqCopy.Imp[0].TagID = admanExt.TagID + + adapterReq, errors := a.makeRequest(&reqCopy) + if adapterReq != nil { + adapterRequests = append(adapterRequests, adapterReq) + } + errs = append(errs, errors...) + } + return adapterRequests, errs +} + +func (a *AdmanAdapter) makeRequest(request *openrtb.BidRequest) (*adapters.RequestData, []error) { + + var errs []error + + reqJSON, err := json.Marshal(request) + + if err != nil { + errs = append(errs, err) + return nil, errs + } + + headers := http.Header{} + headers.Add("Content-Type", "application/json;charset=utf-8") + headers.Add("Accept", "application/json") + return &adapters.RequestData{ + Method: "POST", + Uri: a.URI, + Body: reqJSON, + Headers: headers, + }, errs +} + +// MakeBids makes the bids +func (a *AdmanAdapter) MakeBids(internalRequest *openrtb.BidRequest, externalRequest *adapters.RequestData, response *adapters.ResponseData) (*adapters.BidderResponse, []error) { + var errs []error + + if response.StatusCode == http.StatusNoContent { + return nil, nil + } + + if response.StatusCode == http.StatusNotFound { + return nil, []error{&errortypes.BadServerResponse{ + Message: fmt.Sprintf("Unexpected status code: %d. Run with request.debug = 1 for more info", response.StatusCode), + }} + } + + var bidResp openrtb.BidResponse + + if err := json.Unmarshal(response.Body, &bidResp); err != nil { + return nil, []error{err} + } + + bidResponse := adapters.NewBidderResponseWithBidsCapacity(1) + + for _, sb := range bidResp.SeatBid { + for i := range sb.Bid { + bidType, err := getMediaTypeForImp(sb.Bid[i].ImpID, internalRequest.Imp) + if err != nil { + errs = append(errs, err) + } else { + b := &adapters.TypedBid{ + Bid: &sb.Bid[i], + BidType: bidType, + } + bidResponse.Bids = append(bidResponse.Bids, b) + } + } + } + return bidResponse, errs +} + +func getMediaTypeForImp(impID string, imps []openrtb.Imp) (openrtb_ext.BidType, error) { + mediaType := openrtb_ext.BidTypeBanner + for _, imp := range imps { + if imp.ID == impID { + if imp.Banner == nil && imp.Video != nil { + mediaType = openrtb_ext.BidTypeVideo + } + return mediaType, nil + } + } + + // This shouldnt happen. Lets handle it just incase by returning an error. + return "", &errortypes.BadInput{ + Message: fmt.Sprintf("Failed to find impression \"%s\" ", impID), + } +} diff --git a/adapters/adman/adman_test.go b/adapters/adman/adman_test.go new file mode 100644 index 00000000000..66d84aa8b81 --- /dev/null +++ b/adapters/adman/adman_test.go @@ -0,0 +1,12 @@ +package adman + +import ( + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adapterstest" +) + +func TestJsonSamples(t *testing.T) { + admanAdapter := NewAdmanBidder("http://pub.admanmedia.com/?c=o&m=ortb") + adapterstest.RunJSONBidderTest(t, "admantest", admanAdapter) +} diff --git a/adapters/adman/admantest/exemplary/simple-banner.json b/adapters/adman/admantest/exemplary/simple-banner.json new file mode 100644 index 00000000000..8bbe16aa0fe --- /dev/null +++ b/adapters/adman/admantest/exemplary/simple-banner.json @@ -0,0 +1,134 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "16", + "ext": { + "bidder": { + "TagID": "16" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } +}, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://pub.admanmedia.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "16", + "ext": { + "bidder": { + "TagID": "16" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 300, + "h": 250, + "ext": { + "prebid": { + "type": "banner" + } + } + } + ], + "seat": "adman" + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 300, + "h": 250, + "ext": { + "prebid": { + "type": "banner" + } + } + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/adman/admantest/exemplary/simple-video.json b/adapters/adman/admantest/exemplary/simple-video.json new file mode 100644 index 00000000000..159a30a93e0 --- /dev/null +++ b/adapters/adman/admantest/exemplary/simple-video.json @@ -0,0 +1,119 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + }, + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "imp": [ + { + "id": "test-imp-id", + "video": { + "mimes": ["video/mp4"], + "protocols": [2, 5], + "w": 1024, + "h": 576 + }, + "ext": { + "bidder": { + "TagID": "22" + } + } + } + ] + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://pub.admanmedia.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + }, + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "imp": [ + { + "id": "test-imp-id", + "video": { + "mimes": ["video/mp4"], + "protocols": [2, 5], + "w": 1024, + "h": 576 + }, + "tagid": "22", + "ext": { + "bidder": { + "TagID": "22" + } + } + } + ] + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "00:01:00", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "ext": { + "prebid": { + "type": "video" + } + } + } + ], + "seat": "adman" + } + ], + "cur": "USD" + } + } + } + ], + + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "00:01:00", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "ext": { + "prebid": { + "type": "video" + } + } + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/adman/admantest/exemplary/simple-web-banner.json b/adapters/adman/admantest/exemplary/simple-web-banner.json new file mode 100644 index 00000000000..0ceaac7c6d5 --- /dev/null +++ b/adapters/adman/admantest/exemplary/simple-web-banner.json @@ -0,0 +1,133 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "bidder": { + "TagID": "1" + } + } + } + ], + "site": { + "id": "1", + "domain": "test.com" + }, + "device": { + "ip": "123.123.123.123" + } + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://pub.admanmedia.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "bidder": { + "TagID": "1" + } + } + } + ], + "site": { + "id": "1", + "domain": "test.com" + }, + "device": { + "ip": "123.123.123.123" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 468, + "h": 60, + "ext": { + "prebid": { + "type": "banner" + } + } + } + ], + "seat": "adman" + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 468, + "h": 60, + "ext": { + "prebid": { + "type": "banner" + } + } + }, + "type": "banner" + } + ] + } + ] + } + \ No newline at end of file diff --git a/adapters/adman/admantest/params/banner.json b/adapters/adman/admantest/params/banner.json new file mode 100644 index 00000000000..03fa8f3f2d8 --- /dev/null +++ b/adapters/adman/admantest/params/banner.json @@ -0,0 +1,3 @@ +{ + "TagID": "16" +} \ No newline at end of file diff --git a/adapters/adman/admantest/params/race/banner.json b/adapters/adman/admantest/params/race/banner.json new file mode 100644 index 00000000000..03fa8f3f2d8 --- /dev/null +++ b/adapters/adman/admantest/params/race/banner.json @@ -0,0 +1,3 @@ +{ + "TagID": "16" +} \ No newline at end of file diff --git a/adapters/adman/admantest/params/race/video.json b/adapters/adman/admantest/params/race/video.json new file mode 100644 index 00000000000..e776c928a7e --- /dev/null +++ b/adapters/adman/admantest/params/race/video.json @@ -0,0 +1,3 @@ +{ + "TagID": "22" +} \ No newline at end of file diff --git a/adapters/adman/admantest/params/video.json b/adapters/adman/admantest/params/video.json new file mode 100644 index 00000000000..e776c928a7e --- /dev/null +++ b/adapters/adman/admantest/params/video.json @@ -0,0 +1,3 @@ +{ + "TagID": "22" +} \ No newline at end of file diff --git a/adapters/adman/admantest/supplemental/bad-imp-ext.json b/adapters/adman/admantest/supplemental/bad-imp-ext.json new file mode 100644 index 00000000000..db3c8de5767 --- /dev/null +++ b/adapters/adman/admantest/supplemental/bad-imp-ext.json @@ -0,0 +1,42 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "16", + "ext": { + "adman": { + "TagID": "16" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } +}, +"expectedMakeRequestsErrors": [ + { + "value": "unexpected end of JSON input", + "comparison": "literal" + } +] +} diff --git a/adapters/adman/admantest/supplemental/bad_response.json b/adapters/adman/admantest/supplemental/bad_response.json new file mode 100644 index 00000000000..d5a28c74256 --- /dev/null +++ b/adapters/adman/admantest/supplemental/bad_response.json @@ -0,0 +1,85 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pub.admanmedia.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + } + }, + "mockResponse": { + "status": 200, + "body": "" + } + }], + "expectedMakeBidsErrors": [ + { + "value": "json: cannot unmarshal string into Go value of type openrtb.BidResponse", + "comparison": "literal" + } + ] +} diff --git a/adapters/adman/admantest/supplemental/no-imp-ext-1.json b/adapters/adman/admantest/supplemental/no-imp-ext-1.json new file mode 100644 index 00000000000..8fad5ba5ef0 --- /dev/null +++ b/adapters/adman/admantest/supplemental/no-imp-ext-1.json @@ -0,0 +1,39 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "16", + "ext": "" + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } + }, + "expectedMakeRequestsErrors": [ + { + "value": "json: cannot unmarshal string into Go value of type adapters.ExtImpBidder", + "comparison": "literal" + } + ] + } + \ No newline at end of file diff --git a/adapters/adman/admantest/supplemental/no-imp-ext-2.json b/adapters/adman/admantest/supplemental/no-imp-ext-2.json new file mode 100644 index 00000000000..337dfd044b3 --- /dev/null +++ b/adapters/adman/admantest/supplemental/no-imp-ext-2.json @@ -0,0 +1,39 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "16", + "ext": {} + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } + }, + "expectedMakeRequestsErrors": [ + { + "value": "unexpected end of JSON input", + "comparison": "literal" + } + ] + } + \ No newline at end of file diff --git a/adapters/adman/admantest/supplemental/status-204.json b/adapters/adman/admantest/supplemental/status-204.json new file mode 100644 index 00000000000..72b28bffdcf --- /dev/null +++ b/adapters/adman/admantest/supplemental/status-204.json @@ -0,0 +1,79 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pub.admanmedia.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + } + }, + "mockResponse": { + "status": 204, + "body": {} + } + }] +} diff --git a/adapters/adman/admantest/supplemental/status-404.json b/adapters/adman/admantest/supplemental/status-404.json new file mode 100644 index 00000000000..043afbdc1dc --- /dev/null +++ b/adapters/adman/admantest/supplemental/status-404.json @@ -0,0 +1,85 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "100000000", + "ext": { + "bidder": { + "TagID": "100000000" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pub.admanmedia.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "100000000", + "ext": { + "bidder": { + "TagID": "100000000" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + } + }, + "mockResponse": { + "status": 404, + "body": {} + } + }], + "expectedMakeBidsErrors": [ + { + "value": "Unexpected status code: 404. Run with request.debug = 1 for more info", + "comparison": "literal" + } + ] +} diff --git a/adapters/adman/params_test.go b/adapters/adman/params_test.go new file mode 100644 index 00000000000..4cea67cc098 --- /dev/null +++ b/adapters/adman/params_test.go @@ -0,0 +1,46 @@ +package adman + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" +) + +// TestValidParams makes sure that the adman schema accepts all imp.ext fields which we intend to support. +func TestValidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + if err != nil { + t.Fatalf("Failed to fetch the json-schemas. %v", err) + } + + for _, validParam := range validParams { + if err := validator.Validate(openrtb_ext.BidderAdman, json.RawMessage(validParam)); err != nil { + t.Errorf("Schema rejected adman params: %s", validParam) + } + } +} + +// TestInvalidParams makes sure that the adman schema rejects all the imp.ext fields we don't support. +func TestInvalidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + if err != nil { + t.Fatalf("Failed to fetch the json-schemas. %v", err) + } + + for _, invalidParam := range invalidParams { + if err := validator.Validate(openrtb_ext.BidderAdman, json.RawMessage(invalidParam)); err == nil { + t.Errorf("Schema allowed unexpected params: %s", invalidParam) + } + } +} + +var validParams = []string{ + `{"TagID": "16"}`, +} + +var invalidParams = []string{ + `{"id": "123"}`, + `{"tagid": "123"}`, + `{"TagID": 16}`, +} diff --git a/adapters/adman/usersync.go b/adapters/adman/usersync.go new file mode 100644 index 00000000000..f7edd8c5b70 --- /dev/null +++ b/adapters/adman/usersync.go @@ -0,0 +1,13 @@ +package adman + +import ( + "text/template" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/usersync" +) + +// NewAdmanSyncer returns adman syncer +func NewAdmanSyncer(temp *template.Template) usersync.Usersyncer { + return adapters.NewSyncer("adman", 149, temp, adapters.SyncTypeRedirect) +} diff --git a/adapters/adman/usersync_test.go b/adapters/adman/usersync_test.go new file mode 100644 index 00000000000..db67499e91d --- /dev/null +++ b/adapters/adman/usersync_test.go @@ -0,0 +1,35 @@ +package adman + +import ( + "testing" + "text/template" + + "github.com/PubMatic-OpenWrap/prebid-server/privacy" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" + "github.com/stretchr/testify/assert" +) + +func TestAdmanSyncer(t *testing.T) { + syncURL := "https://sync.admanmedia.com/pbs.gif?gdpr={{.GDPR}}&gdpr_consent={{.GDPRConsent}}&us_privacy={{.USPrivacy}}&redir=http%3A%2F%2Flocalhost%3A8000%2Fsetuid%3Fbidder%3Dadman%26uid%3D%5BUID%5D" + syncURLTemplate := template.Must( + template.New("sync-template").Parse(syncURL), + ) + + syncer := NewAdmanSyncer(syncURLTemplate) + syncInfo, err := syncer.GetUsersyncInfo(privacy.Policies{ + GDPR: gdpr.Policy{ + Signal: "0", + Consent: "ANDFJDS", + }, + CCPA: ccpa.Policy{ + Consent: "1-YY", + }, + }) + + assert.NoError(t, err) + assert.Equal(t, "https://sync.admanmedia.com/pbs.gif?gdpr=0&gdpr_consent=ANDFJDS&us_privacy=1-YY&redir=http%3A%2F%2Flocalhost%3A8000%2Fsetuid%3Fbidder%3Dadman%26uid%3D%5BUID%5D", syncInfo.URL) + assert.Equal(t, "redirect", syncInfo.Type) + assert.EqualValues(t, 149, syncer.GDPRVendorID()) + assert.Equal(t, false, syncInfo.SupportCORS) +} diff --git a/adapters/admixer/usersync_test.go b/adapters/admixer/usersync_test.go index 8903bf89da1..cffe596cdce 100644 --- a/adapters/admixer/usersync_test.go +++ b/adapters/admixer/usersync_test.go @@ -1,12 +1,13 @@ package admixer import ( + "testing" + "text/template" + "github.com/PubMatic-OpenWrap/prebid-server/privacy" "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" "github.com/stretchr/testify/assert" - "testing" - "text/template" ) func TestAdmixerSyncer(t *testing.T) { @@ -22,7 +23,7 @@ func TestAdmixerSyncer(t *testing.T) { Consent: "B", }, CCPA: ccpa.Policy{ - Value: "C", + Consent: "C", }, }) diff --git a/adapters/adoppler/adoppler.go b/adapters/adoppler/adoppler.go index b37aa051363..717ad6211d1 100644 --- a/adapters/adoppler/adoppler.go +++ b/adapters/adoppler/adoppler.go @@ -6,13 +6,18 @@ import ( "fmt" "net/http" "net/url" + "text/template" "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/adapters" "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/macros" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/golang/glog" ) +const DefaultClient = "app" + var bidHeaders http.Header = map[string][]string{ "Accept": {"application/json"}, "Content-Type": {"application/json;charset=utf-8"}, @@ -28,11 +33,17 @@ type adsImpExt struct { } type AdopplerAdapter struct { - endpoint string + endpoint *template.Template } func NewAdopplerBidder(endpoint string) *AdopplerAdapter { - return &AdopplerAdapter{endpoint} + t, err := template.New("endpoint").Parse(endpoint) + if err != nil { + glog.Fatalf("Unable to parse endpoint url template: %s", err) + return nil + } + + return &AdopplerAdapter{t} } func (ads *AdopplerAdapter) MakeRequests( @@ -65,8 +76,13 @@ func (ads *AdopplerAdapter) MakeRequests( continue } - uri := fmt.Sprintf("%s/processHeaderBid/%s", - ads.endpoint, url.PathEscape(ext.AdUnit)) + uri, err := ads.bidUri(ext) + if err != nil { + e := fmt.Sprintf("Unable to build bid URI: %s", + err.Error()) + errs = append(errs, &errortypes.BadInput{e}) + continue + } data := &adapters.RequestData{ Method: "POST", Uri: uri, @@ -172,6 +188,18 @@ func (ads *AdopplerAdapter) MakeBids( return adsResp, nil } +func (ads *AdopplerAdapter) bidUri(ext *openrtb_ext.ExtImpAdoppler) (string, error) { + params := macros.EndpointTemplateParams{} + params.AdUnit = url.PathEscape(ext.AdUnit) + if ext.Client == "" { + params.AccountID = DefaultClient + } else { + params.AccountID = url.PathEscape(ext.Client) + } + + return macros.ResolveMacros(*ads.endpoint, params) +} + func unmarshalExt(ext json.RawMessage) (*openrtb_ext.ExtImpAdoppler, error) { var bext adapters.ExtImpBidder err := json.Unmarshal(ext, &bext) diff --git a/adapters/adoppler/adoppler_test.go b/adapters/adoppler/adoppler_test.go index c3287ed4adb..775524b171d 100644 --- a/adapters/adoppler/adoppler_test.go +++ b/adapters/adoppler/adoppler_test.go @@ -7,6 +7,6 @@ import ( ) func TestJsonSamples(t *testing.T) { - bidder := NewAdopplerBidder("http://adoppler.com") + bidder := NewAdopplerBidder("http://{{.AccountID}}.trustedmarketplace.com/processHeaderBid/{{.AdUnit}}") adapterstest.RunJSONBidderTest(t, "adopplertest", bidder) } diff --git a/adapters/adoppler/adopplertest/exemplary/custom-client.json b/adapters/adoppler/adopplertest/exemplary/custom-client.json new file mode 100644 index 00000000000..6bb32f71546 --- /dev/null +++ b/adapters/adoppler/adopplertest/exemplary/custom-client.json @@ -0,0 +1,80 @@ +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1", + "client":"client1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://client1.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1", + "client":"client1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-unit1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-unit1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"An ad" + } + ] + } + ], + "cur":"USD" + } + } + } + ], + "expectedBidResponses":[ + { + "currency":"USD", + "bids":[ + { + "bid":{ + "id":"req1-unit1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"An ad" + }, + "type":"banner" + } + ] + } + ] +} diff --git a/adapters/adoppler/adopplertest/exemplary/default-client.json b/adapters/adoppler/adopplertest/exemplary/default-client.json new file mode 100644 index 00000000000..25fb71970e0 --- /dev/null +++ b/adapters/adoppler/adopplertest/exemplary/default-client.json @@ -0,0 +1,78 @@ +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-unit1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-unit1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"An ad" + } + ] + } + ], + "cur":"USD" + } + } + } + ], + "expectedBidResponses":[ + { + "currency":"USD", + "bids":[ + { + "bid":{ + "id":"req1-unit1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"An ad" + }, + "type":"banner" + } + ] + } + ] +} diff --git a/adapters/adoppler/adopplertest/exemplary/multibid.json b/adapters/adoppler/adopplertest/exemplary/multibid.json deleted file mode 100644 index 851f4c5b917..00000000000 --- a/adapters/adoppler/adopplertest/exemplary/multibid.json +++ /dev/null @@ -1,60 +0,0 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}, - {"id": "imp2", - "video": {"minduration": 120, - "mimes": ["video/mp4"]}, - "ext": {"bidder": {"adunit": "unit2"}}}, - {"id": "imp3", - "native": {"request": "{}"}, - "ext": {"bidder": {"adunit": "unit3"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp1-resp1", - "seatbid": [{"bid": [{"id": "req1-imp1-bid1", - "impid": "imp1", - "price": 0.12, - "adm": "a banner"}]}], - "cur": "USD"}}}, - {"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit2", - "body": {"id": "req1-unit2", - "imp": [{"id": "imp2", - "video": {"minduration": 120, - "mimes": ["video/mp4"]}, - "ext": {"bidder": {"adunit": "unit2"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp2-resp2", - "seatbid": [{"bid": [{"id": "req1-imp2-bid1", - "impid": "imp2", - "price": 0.24, - "adm": "", - "cat": ["IAB1", "IAB2"], - "ext": {"ads": {"video": {"duration": 121}}}}]}], - "cur": "USD"}}}, - {"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit3", - "body": {"id": "req1-unit3", - "imp": [{"id": "imp3", - "native": {"request": "{}"}, - "ext": {"bidder": {"adunit": "unit3"}}}]}}, - "mockResponse": {"status": 204, - "body": ""}}], - "expectedBidResponses": [{"currency": "USD", - "bids": [{"bid": {"id": "req1-imp1-bid1", - "impid": "imp1", - "price": 0.12, - "adm": "a banner"}, - "type": "banner"}]}, - {"currency": "USD", - "bids": [{"bid": {"id": "req1-imp2-bid1", - "impid": "imp2", - "price": 0.24, - "adm": "", - "cat": ["IAB1", "IAB2"], - "ext": {"ads": {"video": {"duration": 121}}}}, - "type": "video"}]}]} diff --git a/adapters/adoppler/adopplertest/exemplary/multiimp.json b/adapters/adoppler/adopplertest/exemplary/multiimp.json new file mode 100644 index 00000000000..6eebbe43071 --- /dev/null +++ b/adapters/adoppler/adopplertest/exemplary/multiimp.json @@ -0,0 +1,207 @@ +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + }, + { + "id":"imp2", + "video":{ + "minduration":120, + "mimes":[ + "video/mp4" + ] + }, + "ext":{ + "bidder":{ + "adunit":"unit2" + } + } + }, + { + "id":"imp3", + "native":{ + "request":"{}" + }, + "ext":{ + "bidder":{ + "adunit":"unit3" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp1-resp1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"a banner" + } + ] + } + ], + "cur":"USD" + } + } + }, + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit2", + "body":{ + "id":"req1-unit2", + "imp":[ + { + "id":"imp2", + "video":{ + "minduration":120, + "mimes":[ + "video/mp4" + ] + }, + "ext":{ + "bidder":{ + "adunit":"unit2" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp2-resp2", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp2-bid1", + "impid":"imp2", + "price":0.24, + "adm":"", + "cat":[ + "IAB1", + "IAB2" + ], + "ext":{ + "ads":{ + "video":{ + "duration":121 + } + } + } + } + ] + } + ], + "cur":"USD" + } + } + }, + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit3", + "body":{ + "id":"req1-unit3", + "imp":[ + { + "id":"imp3", + "native":{ + "request":"{}" + }, + "ext":{ + "bidder":{ + "adunit":"unit3" + } + } + } + ] + } + }, + "mockResponse":{ + "status":204, + "body":"" + } + } + ], + "expectedBidResponses":[ + { + "currency":"USD", + "bids":[ + { + "bid":{ + "id":"req1-imp1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"a banner" + }, + "type":"banner" + } + ] + }, + { + "currency":"USD", + "bids":[ + { + "bid":{ + "id":"req1-imp2-bid1", + "impid":"imp2", + "price":0.24, + "adm":"", + "cat":[ + "IAB1", + "IAB2" + ], + "ext":{ + "ads":{ + "video":{ + "duration":121 + } + } + } + }, + "type":"video" + } + ] + } + ] +} diff --git a/adapters/adoppler/adopplertest/exemplary/no-bid.json b/adapters/adoppler/adopplertest/exemplary/no-bid.json deleted file mode 100644 index 0e0f13586a8..00000000000 --- a/adapters/adoppler/adopplertest/exemplary/no-bid.json +++ /dev/null @@ -1,13 +0,0 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 204, - "body": ""}}], - "expectedBidResponses": []} diff --git a/adapters/adoppler/adopplertest/supplemental/bad-request.json b/adapters/adoppler/adopplertest/supplemental/bad-request.json index 3bdd5a5544e..ae515e01e18 100644 --- a/adapters/adoppler/adopplertest/supplemental/bad-request.json +++ b/adapters/adoppler/adopplertest/supplemental/bad-request.json @@ -1,15 +1,56 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 400, - "body": ""}}], - "expectedBidResponses": [], - "expectedMakeBidsErrors": [{"value": "bad request", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":400, + "body":"" + } + } + ], + "expectedBidResponses":[ + + ], + "expectedMakeBidsErrors":[ + { + "value":"bad request", + "comparison":"literal" + } + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/duplicate-imp.json b/adapters/adoppler/adopplertest/supplemental/duplicate-imp.json index 4382e36c54e..d6f17a6bb2c 100644 --- a/adapters/adoppler/adopplertest/supplemental/duplicate-imp.json +++ b/adapters/adoppler/adopplertest/supplemental/duplicate-imp.json @@ -1,38 +1,128 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}, - {"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit2"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp1-resp1", - "seatbid": [{"bid": [{"id": "req1-imp1-bid1", - "impid": "imp1", - "price": 0.12, - "adm": "a banner"}]}], - "cur": "USD"}}}, - {"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit2", - "body": {"id": "req1-unit2", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit2"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp1-resp1", - "seatbid": [{"bid": [{"id": "req1-imp1-bid1", - "impid": "imp1", - "price": 0.12, - "adm": "a banner"}]}], - "cur": "USD"}}}], - "expectedBidResponses": [], - "expectedMakeBidsErrors": [{"value": "duplicate $.imp.id imp1", - "comparison": "literal"}, - {"value": "duplicate $.imp.id imp1", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + }, + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit2" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp1-resp1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"a banner" + } + ] + } + ], + "cur":"USD" + } + } + }, + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit2", + "body":{ + "id":"req1-unit2", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit2" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp1-resp1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp1-bid1", + "impid":"imp1", + "price":0.12, + "adm":"a banner" + } + ] + } + ], + "cur":"USD" + } + } + } + ], + "expectedBidResponses":[ + + ], + "expectedMakeBidsErrors":[ + { + "value":"duplicate $.imp.id imp1", + "comparison":"literal" + }, + { + "value":"duplicate $.imp.id imp1", + "comparison":"literal" + } + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/invalid-impid.json b/adapters/adoppler/adopplertest/supplemental/invalid-impid.json index 2e6ecf4a96c..b5f062ac94a 100644 --- a/adapters/adoppler/adopplertest/supplemental/invalid-impid.json +++ b/adapters/adoppler/adopplertest/supplemental/invalid-impid.json @@ -1,20 +1,71 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp1-resp1", - "seatbid": [{"bid": [{"id": "req1-imp1-bid1", - "impid": "invalid", - "price": 0.12, - "adm": "a banner"}]}], - "cur": "USD"}}}], - "expectedBidResponses": [], - "expectedMakeBidsErrors": [{"value": "unknown impid: invalid", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp1-resp1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp1-bid1", + "impid":"invalid", + "price":0.12, + "adm":"a banner" + } + ] + } + ], + "cur":"USD" + } + } + } + ], + "expectedBidResponses":[ + + ], + "expectedMakeBidsErrors":[ + { + "value":"unknown impid: invalid", + "comparison":"literal" + } + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/invalid-response.json b/adapters/adoppler/adopplertest/supplemental/invalid-response.json index 72420881aec..d0a7d2ef84b 100644 --- a/adapters/adoppler/adopplertest/supplemental/invalid-response.json +++ b/adapters/adoppler/adopplertest/supplemental/invalid-response.json @@ -1,15 +1,56 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 200, - "body": "invalid-json"}}], - "expectedBidResponses": [], - "expectedMakeBidsErrors": [{"value": "invalid body: json: cannot unmarshal string into Go value of type openrtb.BidResponse", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":"invalid-json" + } + } + ], + "expectedBidResponses":[ + + ], + "expectedMakeBidsErrors":[ + { + "value":"invalid body: json: cannot unmarshal string into Go value of type openrtb.BidResponse", + "comparison":"literal" + } + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/invalid-video-ext.json b/adapters/adoppler/adopplertest/supplemental/invalid-video-ext.json index d9cb6daa55d..c08cdca5cee 100644 --- a/adapters/adoppler/adopplertest/supplemental/invalid-video-ext.json +++ b/adapters/adoppler/adopplertest/supplemental/invalid-video-ext.json @@ -1,43 +1,145 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "video": {"minduration": 120, - "mimes": ["video/mp4"]}, - "ext": {"bidder": {"adunit": "unit1"}}}, - {"id": "imp2", - "video": {"minduration": 120, - "mimes": ["video/mp4"]}, - "ext": {"bidder": {"adunit": "unit2"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "video": {"minduration": 120, - "mimes": ["video/mp4"]}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp1-resp1", - "seatbid": [{"bid": [{"id": "req1-imp1-bid1", - "impid": "imp1", - "price": 0.24, - "adm": "", - "cat": ["IAB1", "IAB2"], - "ext": {}}]}], - "cur": "USD"}}}, - {"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit2", - "body": {"id": "req1-unit2", - "imp": [{"id": "imp2", - "video": {"minduration": 120, - "mimes": ["video/mp4"]}, - "ext": {"bidder": {"adunit": "unit2"}}}]}}, - "mockResponse": {"status": 200, - "body": {"id": "req1-imp2-resp2", - "seatbid": [{"bid": [{"id": "req1-imp2-bid2", - "impid": "imp2", - "price": 0.24, - "adm": "", - "cat": ["IAB1", "IAB2"], - "ext": ""}]}], - "cur": "USD"}}}], - "expectedMakeBidsErrors": [{"value": "$.seatbid.bid.ext.ads.video required", - "comparison": "literal"}, - {"value": "json: cannot unmarshal string into Go value of type struct { Ads *adoppler.adsImpExt \"json:\\\"ads\\\"\" }", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "video":{ + "minduration":120, + "mimes":[ + "video/mp4" + ] + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + }, + { + "id":"imp2", + "video":{ + "minduration":120, + "mimes":[ + "video/mp4" + ] + }, + "ext":{ + "bidder":{ + "adunit":"unit2" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "video":{ + "minduration":120, + "mimes":[ + "video/mp4" + ] + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp1-resp1", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp1-bid1", + "impid":"imp1", + "price":0.24, + "adm":"", + "cat":[ + "IAB1", + "IAB2" + ], + "ext":{ + + } + } + ] + } + ], + "cur":"USD" + } + } + }, + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit2", + "body":{ + "id":"req1-unit2", + "imp":[ + { + "id":"imp2", + "video":{ + "minduration":120, + "mimes":[ + "video/mp4" + ] + }, + "ext":{ + "bidder":{ + "adunit":"unit2" + } + } + } + ] + } + }, + "mockResponse":{ + "status":200, + "body":{ + "id":"req1-imp2-resp2", + "seatbid":[ + { + "bid":[ + { + "id":"req1-imp2-bid2", + "impid":"imp2", + "price":0.24, + "adm":"", + "cat":[ + "IAB1", + "IAB2" + ], + "ext":"" + } + ] + } + ], + "cur":"USD" + } + } + } + ], + "expectedMakeBidsErrors":[ + { + "value":"$.seatbid.bid.ext.ads.video required", + "comparison":"literal" + }, + { + "value":"json: cannot unmarshal string into Go value of type struct { Ads *adoppler.adsImpExt \"json:\\\"ads\\\"\" }", + "comparison":"literal" + } + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/missing-adunit.json b/adapters/adoppler/adopplertest/supplemental/missing-adunit.json index 82a6a95ed58..df9bbe5771d 100644 --- a/adapters/adoppler/adopplertest/supplemental/missing-adunit.json +++ b/adapters/adoppler/adopplertest/supplemental/missing-adunit.json @@ -1,9 +1,31 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {}}}]}, - "httpCalls": [], - "expectedBidResponses": [], - "expectedMakeRequestsErrors": [{"value": "$.imp.ext.adoppler.adunit required", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + + } + } + } + ] + }, + "httpCalls":[ + + ], + "expectedBidResponses":[ + + ], + "expectedMakeRequestsErrors":[ + { + "value":"$.imp.ext.adoppler.adunit required", + "comparison":"literal" + } + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/no-bid.json b/adapters/adoppler/adopplertest/supplemental/no-bid.json new file mode 100644 index 00000000000..08a29481350 --- /dev/null +++ b/adapters/adoppler/adopplertest/supplemental/no-bid.json @@ -0,0 +1,50 @@ +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":204, + "body":"" + } + } + ], + "expectedBidResponses":[ + + ] +} diff --git a/adapters/adoppler/adopplertest/supplemental/server-error.json b/adapters/adoppler/adopplertest/supplemental/server-error.json index df23bac07df..604b83e74a6 100644 --- a/adapters/adoppler/adopplertest/supplemental/server-error.json +++ b/adapters/adoppler/adopplertest/supplemental/server-error.json @@ -1,15 +1,56 @@ -{"mockBidRequest": {"id": "req1", - "imp":[{"id": "imp1", - "banner": {"w": 100, - "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}, - "httpCalls": [{"expectedRequest": {"uri": "http://adoppler.com/processHeaderBid/unit1", - "body": {"id": "req1-unit1", - "imp": [{"id": "imp1", - "banner": {"w": 100, "h": 200}, - "ext": {"bidder": {"adunit": "unit1"}}}]}}, - "mockResponse": {"status": 500, - "body": ""}}], - "expectedBidResponses": [], - "expectedMakeBidsErrors": [{"value": "unexpected status: 500", - "comparison": "literal"}]} +{ + "mockBidRequest":{ + "id":"req1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + }, + "httpCalls":[ + { + "expectedRequest":{ + "uri":"http://app.trustedmarketplace.com/processHeaderBid/unit1", + "body":{ + "id":"req1-unit1", + "imp":[ + { + "id":"imp1", + "banner":{ + "w":100, + "h":200 + }, + "ext":{ + "bidder":{ + "adunit":"unit1" + } + } + } + ] + } + }, + "mockResponse":{ + "status":500, + "body":"" + } + } + ], + "expectedBidResponses":[ + + ], + "expectedMakeBidsErrors":[ + { + "value":"unexpected status: 500", + "comparison":"literal" + } + ] +} diff --git a/adapters/adpone/usersync.go b/adapters/adpone/usersync.go index b80ee4442a3..63f616091e2 100644 --- a/adapters/adpone/usersync.go +++ b/adapters/adpone/usersync.go @@ -7,7 +7,7 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/usersync" ) -const adponeGDPRVendorID = uint16(16) +const adponeGDPRVendorID = uint16(799) const adponeFamilyName = "adpone" func NewadponeSyncer(urlTemplate *template.Template) usersync.Usersyncer { diff --git a/adapters/adprime/adprime.go b/adapters/adprime/adprime.go new file mode 100644 index 00000000000..2ef5f26edc7 --- /dev/null +++ b/adapters/adprime/adprime.go @@ -0,0 +1,138 @@ +package adprime + +import ( + "encoding/json" + "fmt" + "net/http" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/buger/jsonparser" +) + +// AdprimeAdapter struct +type AdprimeAdapter struct { + URI string +} + +// NewAdprimeBidder Initializes the Bidder +func NewAdprimeBidder(endpoint string) *AdprimeAdapter { + return &AdprimeAdapter{ + URI: endpoint, + } +} + +// MakeRequests create bid request for adprime demand +func (a *AdprimeAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *adapters.ExtraRequestInfo) ([]*adapters.RequestData, []error) { + var errs []error + var err error + var tagID string + + var adapterRequests []*adapters.RequestData + + reqCopy := *request + for _, imp := range request.Imp { + reqCopy.Imp = []openrtb.Imp{imp} + + tagID, err = jsonparser.GetString(reqCopy.Imp[0].Ext, "bidder", "TagID") + if err != nil { + errs = append(errs, err) + continue + } + + reqCopy.Imp[0].TagID = tagID + + adapterReq, errors := a.makeRequest(&reqCopy) + if adapterReq != nil { + adapterRequests = append(adapterRequests, adapterReq) + } + errs = append(errs, errors...) + } + return adapterRequests, errs +} + +func (a *AdprimeAdapter) makeRequest(request *openrtb.BidRequest) (*adapters.RequestData, []error) { + + var errs []error + + reqJSON, err := json.Marshal(request) + + if err != nil { + errs = append(errs, err) + return nil, errs + } + + headers := http.Header{} + headers.Add("Content-Type", "application/json;charset=utf-8") + headers.Add("Accept", "application/json") + return &adapters.RequestData{ + Method: "POST", + Uri: a.URI, + Body: reqJSON, + Headers: headers, + }, errs +} + +// MakeBids makes the bids +func (a *AdprimeAdapter) MakeBids(internalRequest *openrtb.BidRequest, externalRequest *adapters.RequestData, response *adapters.ResponseData) (*adapters.BidderResponse, []error) { + var errs []error + + if response.StatusCode == http.StatusNoContent { + return nil, nil + } + + if response.StatusCode == http.StatusNotFound { + return nil, []error{&errortypes.BadServerResponse{ + Message: fmt.Sprintf("Page not found: %d. Run with request.debug = 1 for more info", response.StatusCode), + }} + } + + if response.StatusCode != http.StatusOK { + return nil, []error{&errortypes.BadServerResponse{ + Message: fmt.Sprintf("Unexpected status code: %d. Run with request.debug = 1 for more info", response.StatusCode), + }} + } + + var bidResp openrtb.BidResponse + + if err := json.Unmarshal(response.Body, &bidResp); err != nil { + return nil, []error{err} + } + + bidResponse := adapters.NewBidderResponseWithBidsCapacity(1) + + for _, sb := range bidResp.SeatBid { + for i := range sb.Bid { + bidType, err := getMediaTypeForImp(sb.Bid[i].ImpID, internalRequest.Imp) + if err != nil { + errs = append(errs, err) + } else { + b := &adapters.TypedBid{ + Bid: &sb.Bid[i], + BidType: bidType, + } + bidResponse.Bids = append(bidResponse.Bids, b) + } + } + } + return bidResponse, errs +} + +func getMediaTypeForImp(impID string, imps []openrtb.Imp) (openrtb_ext.BidType, error) { + mediaType := openrtb_ext.BidTypeBanner + for _, imp := range imps { + if imp.ID == impID { + if imp.Banner == nil && imp.Video != nil { + mediaType = openrtb_ext.BidTypeVideo + } + return mediaType, nil + } + } + + // This shouldnt happen. Lets handle it just incase by returning an error. + return "", &errortypes.BadInput{ + Message: fmt.Sprintf("Failed to find impression \"%s\" ", impID), + } +} diff --git a/adapters/adprime/adprime_test.go b/adapters/adprime/adprime_test.go new file mode 100644 index 00000000000..1da70595401 --- /dev/null +++ b/adapters/adprime/adprime_test.go @@ -0,0 +1,12 @@ +package adprime + +import ( + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adapterstest" +) + +func TestJsonSamples(t *testing.T) { + adprimeAdapter := NewAdprimeBidder("http://delta.adprime.com/?c=o&m=ortb") + adapterstest.RunJSONBidderTest(t, "adprimetest", adprimeAdapter) +} diff --git a/adapters/adprime/adprimetest/exemplary/simple-banner.json b/adapters/adprime/adprimetest/exemplary/simple-banner.json new file mode 100644 index 00000000000..076175c6274 --- /dev/null +++ b/adapters/adprime/adprimetest/exemplary/simple-banner.json @@ -0,0 +1,134 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "bidder": { + "TagID": "1" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } +}, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://delta.adprime.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "bidder": { + "TagID": "1" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 300, + "h": 250, + "ext": { + "prebid": { + "type": "banner" + } + } + } + ], + "seat": "adprime" + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 300, + "h": 250, + "ext": { + "prebid": { + "type": "banner" + } + } + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/adprime/adprimetest/exemplary/simple-video.json b/adapters/adprime/adprimetest/exemplary/simple-video.json new file mode 100644 index 00000000000..3e61c4dddd1 --- /dev/null +++ b/adapters/adprime/adprimetest/exemplary/simple-video.json @@ -0,0 +1,119 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + }, + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "imp": [ + { + "id": "test-imp-id", + "video": { + "mimes": ["video/mp4"], + "protocols": [2, 5], + "w": 1024, + "h": 576 + }, + "ext": { + "bidder": { + "TagID": "288" + } + } + } + ] + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://delta.adprime.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + }, + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "imp": [ + { + "id": "test-imp-id", + "video": { + "mimes": ["video/mp4"], + "protocols": [2, 5], + "w": 1024, + "h": 576 + }, + "tagid": "288", + "ext": { + "bidder": { + "TagID": "288" + } + } + } + ] + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "00:01:00", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "ext": { + "prebid": { + "type": "video" + } + } + } + ], + "seat": "adprime" + } + ], + "cur": "USD" + } + } + } + ], + + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "00:01:00", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "ext": { + "prebid": { + "type": "video" + } + } + }, + "type": "video" + } + ] + } + ] +} diff --git a/adapters/adprime/adprimetest/exemplary/simple-web-banner.json b/adapters/adprime/adprimetest/exemplary/simple-web-banner.json new file mode 100644 index 00000000000..a955854fb31 --- /dev/null +++ b/adapters/adprime/adprimetest/exemplary/simple-web-banner.json @@ -0,0 +1,133 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "bidder": { + "TagID": "1" + } + } + } + ], + "site": { + "id": "1", + "domain": "test.com" + }, + "device": { + "ip": "123.123.123.123" + } + }, + + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://delta.adprime.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "bidder": { + "TagID": "1" + } + } + } + ], + "site": { + "id": "1", + "domain": "test.com" + }, + "device": { + "ip": "123.123.123.123" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 468, + "h": 60, + "ext": { + "prebid": { + "type": "banner" + } + } + } + ], + "seat": "adprime" + } + ], + "cur": "USD" + } + } + } + ], + + "expectedBidResponses": [ + { + "bids":[ + { + "bid": { + "id": "test_bid_id", + "impid": "test-imp-id", + "price": 0.27543, + "adm": "", + "cid": "test_cid", + "crid": "test_crid", + "dealid": "test_dealid", + "w": 468, + "h": 60, + "ext": { + "prebid": { + "type": "banner" + } + } + }, + "type": "banner" + } + ] + } + ] + } + \ No newline at end of file diff --git a/adapters/adprime/adprimetest/params/banner.json b/adapters/adprime/adprimetest/params/banner.json new file mode 100644 index 00000000000..e3f4cb7605a --- /dev/null +++ b/adapters/adprime/adprimetest/params/banner.json @@ -0,0 +1,3 @@ +{ + "TagID": "1" +} \ No newline at end of file diff --git a/adapters/adprime/adprimetest/params/race/banner.json b/adapters/adprime/adprimetest/params/race/banner.json new file mode 100644 index 00000000000..e3f4cb7605a --- /dev/null +++ b/adapters/adprime/adprimetest/params/race/banner.json @@ -0,0 +1,3 @@ +{ + "TagID": "1" +} \ No newline at end of file diff --git a/adapters/adprime/adprimetest/params/race/video.json b/adapters/adprime/adprimetest/params/race/video.json new file mode 100644 index 00000000000..c8d14757903 --- /dev/null +++ b/adapters/adprime/adprimetest/params/race/video.json @@ -0,0 +1,3 @@ +{ + "TagID": "288" +} \ No newline at end of file diff --git a/adapters/adprime/adprimetest/params/video.json b/adapters/adprime/adprimetest/params/video.json new file mode 100644 index 00000000000..c8d14757903 --- /dev/null +++ b/adapters/adprime/adprimetest/params/video.json @@ -0,0 +1,3 @@ +{ + "TagID": "288" +} \ No newline at end of file diff --git a/adapters/adprime/adprimetest/supplemental/bad-imp-ext.json b/adapters/adprime/adprimetest/supplemental/bad-imp-ext.json new file mode 100644 index 00000000000..a95c56e8426 --- /dev/null +++ b/adapters/adprime/adprimetest/supplemental/bad-imp-ext.json @@ -0,0 +1,42 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": { + "adprime": { + "TagID": "1" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } +}, +"expectedMakeRequestsErrors": [ + { + "value": "Key path not found", + "comparison": "literal" + } +] +} diff --git a/adapters/adprime/adprimetest/supplemental/bad_response.json b/adapters/adprime/adprimetest/supplemental/bad_response.json new file mode 100644 index 00000000000..329e9c7269f --- /dev/null +++ b/adapters/adprime/adprimetest/supplemental/bad_response.json @@ -0,0 +1,85 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://delta.adprime.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + } + }, + "mockResponse": { + "status": 200, + "body": "" + } + }], + "expectedMakeBidsErrors": [ + { + "value": "json: cannot unmarshal string into Go value of type openrtb.BidResponse", + "comparison": "literal" + } + ] +} diff --git a/adapters/adprime/adprimetest/supplemental/no-imp-ext-1.json b/adapters/adprime/adprimetest/supplemental/no-imp-ext-1.json new file mode 100644 index 00000000000..1e38dbe4541 --- /dev/null +++ b/adapters/adprime/adprimetest/supplemental/no-imp-ext-1.json @@ -0,0 +1,39 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": "" + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } + }, + "expectedMakeRequestsErrors": [ + { + "value": "Key path not found", + "comparison": "literal" + } + ] + } + \ No newline at end of file diff --git a/adapters/adprime/adprimetest/supplemental/no-imp-ext-2.json b/adapters/adprime/adprimetest/supplemental/no-imp-ext-2.json new file mode 100644 index 00000000000..f9759fae8ff --- /dev/null +++ b/adapters/adprime/adprimetest/supplemental/no-imp-ext-2.json @@ -0,0 +1,39 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "1", + "ext": {} + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "zxcjbzxmc-zxcbmz-zxbcz-zxczx" + } + }, + "expectedMakeRequestsErrors": [ + { + "value": "Key path not found", + "comparison": "literal" + } + ] + } + \ No newline at end of file diff --git a/adapters/adprime/adprimetest/supplemental/status-204.json b/adapters/adprime/adprimetest/supplemental/status-204.json new file mode 100644 index 00000000000..44ee59d4d28 --- /dev/null +++ b/adapters/adprime/adprimetest/supplemental/status-204.json @@ -0,0 +1,79 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://delta.adprime.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "17", + "ext": { + "bidder": { + "TagID": "17" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + } + }, + "mockResponse": { + "status": 204, + "body": {} + } + }] +} diff --git a/adapters/adprime/adprimetest/supplemental/status-404.json b/adapters/adprime/adprimetest/supplemental/status-404.json new file mode 100644 index 00000000000..c2b303f0cb4 --- /dev/null +++ b/adapters/adprime/adprimetest/supplemental/status-404.json @@ -0,0 +1,85 @@ +{ + "mockBidRequest": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "100000000", + "ext": { + "bidder": { + "TagID": "100000000" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://delta.adprime.com/?c=o&m=ortb", + "body": { + "id": "test-request-id", + "imp": [ + { + "id": "test-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "tagid": "100000000", + "ext": { + "bidder": { + "TagID": "100000000" + } + } + } + ], + "app": { + "id": "1", + "bundle": "com.wls.testwlsapplication" + }, + "device": { + "ip": "123.123.123.123", + "ifa": "sdjfksdf-dfsds-dsdg-dsgg" + } + } + }, + "mockResponse": { + "status": 404, + "body": {} + } + }], + "expectedMakeBidsErrors": [ + { + "value": "Page not found: 404. Run with request.debug = 1 for more info", + "comparison": "literal" + } + ] +} diff --git a/adapters/adprime/params_test.go b/adapters/adprime/params_test.go new file mode 100644 index 00000000000..bea13e32c13 --- /dev/null +++ b/adapters/adprime/params_test.go @@ -0,0 +1,46 @@ +package adprime + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" +) + +// TestValidParams makes sure that the adprime schema accepts all imp.ext fields which we intend to support. +func TestValidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + if err != nil { + t.Fatalf("Failed to fetch the json-schemas. %v", err) + } + + for _, validParam := range validParams { + if err := validator.Validate(openrtb_ext.BidderAdprime, json.RawMessage(validParam)); err != nil { + t.Errorf("Schema rejected adprime params: %s", validParam) + } + } +} + +// TestInvalidParams makes sure that the adprime schema rejects all the imp.ext fields we don't support. +func TestInvalidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + if err != nil { + t.Fatalf("Failed to fetch the json-schemas. %v", err) + } + + for _, invalidParam := range invalidParams { + if err := validator.Validate(openrtb_ext.BidderAdprime, json.RawMessage(invalidParam)); err == nil { + t.Errorf("Schema allowed unexpected params: %s", invalidParam) + } + } +} + +var validParams = []string{ + `{"TagID": "1"}`, +} + +var invalidParams = []string{ + `{"id": "123"}`, + `{"tagid": "123"}`, + `{"TagID": 16}`, +} diff --git a/adapters/adtarget/usersync_test.go b/adapters/adtarget/usersync_test.go index ccaf7ee1bf9..419a6cb037e 100644 --- a/adapters/adtarget/usersync_test.go +++ b/adapters/adtarget/usersync_test.go @@ -2,10 +2,11 @@ package adtarget import ( "fmt" - "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" "testing" "text/template" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" + "github.com/PubMatic-OpenWrap/prebid-server/privacy" "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" "github.com/stretchr/testify/assert" @@ -25,7 +26,7 @@ func TestAdtargetSyncer(t *testing.T) { Consent: "123", }, CCPA: ccpa.Policy{ - Value: "1-YY", + Consent: "1-YY", }, }) diff --git a/adapters/adtelligent/usersync.go b/adapters/adtelligent/usersync.go index 4315a6f348c..387c65bb46d 100644 --- a/adapters/adtelligent/usersync.go +++ b/adapters/adtelligent/usersync.go @@ -8,5 +8,5 @@ import ( ) func NewAdtelligentSyncer(temp *template.Template) usersync.Usersyncer { - return adapters.NewSyncer("adtelligent", 0, temp, adapters.SyncTypeRedirect) + return adapters.NewSyncer("adtelligent", 410, temp, adapters.SyncTypeRedirect) } diff --git a/adapters/adtelligent/usersync_test.go b/adapters/adtelligent/usersync_test.go index e0006847ccd..7cc92eb4011 100644 --- a/adapters/adtelligent/usersync_test.go +++ b/adapters/adtelligent/usersync_test.go @@ -25,6 +25,6 @@ func TestAdtelligentSyncer(t *testing.T) { assert.NoError(t, err) assert.Equal(t, "//sync.adtelligent.com/csync?t=p&ep=0&redir=localhost%2Fsetuid%3Fbidder%3Dadtelligent%26gdpr%3D0%26gdpr_consent%3D%26uid%3D%7Buid%7D", syncInfo.URL) assert.Equal(t, "redirect", syncInfo.Type) - assert.EqualValues(t, 0, syncer.GDPRVendorID()) + assert.EqualValues(t, 410, syncer.GDPRVendorID()) assert.Equal(t, false, syncInfo.SupportCORS) } diff --git a/adapters/aja/usersync_test.go b/adapters/aja/usersync_test.go index 54b3ed01212..bf03f47af19 100644 --- a/adapters/aja/usersync_test.go +++ b/adapters/aja/usersync_test.go @@ -1,10 +1,11 @@ package aja import ( - "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" "testing" "text/template" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" + "github.com/PubMatic-OpenWrap/prebid-server/privacy" "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" "github.com/stretchr/testify/assert" @@ -23,7 +24,7 @@ func TestAJASyncer(t *testing.T) { Consent: "BOPVK28OVJoTBABABAENBs-AAAAhuAKAANAAoACwAGgAPAAxAB0AHgAQAAiABOADkA", }, CCPA: ccpa.Policy{ - Value: "C", + Consent: "C", }, }) diff --git a/adapters/amx/amx.go b/adapters/amx/amx.go new file mode 100644 index 00000000000..2578ab786c6 --- /dev/null +++ b/adapters/amx/amx.go @@ -0,0 +1,210 @@ +package amx + +import ( + "encoding/json" + "fmt" + "net/http" + "net/url" + "strings" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/golang/glog" +) + +const vastImpressionFormat = "" +const vastSearchPoint = "" +const nbrHeaderName = "x-nbr" +const adapterVersion = "pbs1.0" + +// AMXAdapter is the AMX bid adapter +type AMXAdapter struct { + endpoint string +} + +// NewAMXBidder creates an AMXAdapter +func NewAMXBidder(endpoint string) *AMXAdapter { + endpointURL, err := url.Parse(endpoint) + if err != nil { + glog.Fatalf("invalid endpoint provided to AMX: %s, error: %v", endpoint, err) + return nil + } + + qs, err := url.ParseQuery(endpointURL.RawQuery) + if err != nil { + glog.Fatalf("invalid query parameters in the endpoint: %s, error: %v", endpointURL.RawQuery, err) + return nil + } + + qs.Add("v", adapterVersion) + endpointURL.RawQuery = qs.Encode() + + return &AMXAdapter{endpoint: endpointURL.String()} +} + +type amxExt struct { + Bidder openrtb_ext.ExtImpAMX `json:"bidder"` +} + +func ensurePublisherWithID(pub *openrtb.Publisher, publisherID string) openrtb.Publisher { + if pub == nil { + return openrtb.Publisher{ID: publisherID} + } + + pubCopy := *pub + pubCopy.ID = publisherID + return pubCopy +} + +// MakeRequests creates AMX adapter requests +func (adapter *AMXAdapter) MakeRequests(request *openrtb.BidRequest, req *adapters.ExtraRequestInfo) (reqsBidder []*adapters.RequestData, errs []error) { + reqCopy := *request + + var publisherID string + for idx, imp := range reqCopy.Imp { + var params amxExt + if err := json.Unmarshal(imp.Ext, ¶ms); err == nil { + if params.Bidder.TagID != "" { + publisherID = params.Bidder.TagID + } + + // if it has an adUnitId, set as the tagid + if params.Bidder.AdUnitID != "" { + imp.TagID = params.Bidder.AdUnitID + reqCopy.Imp[idx] = imp + } + } + } + + if publisherID != "" { + if reqCopy.App != nil { + publisher := ensurePublisherWithID(reqCopy.App.Publisher, publisherID) + appCopy := *request.App + appCopy.Publisher = &publisher + reqCopy.App = &appCopy + } + if reqCopy.Site != nil { + publisher := ensurePublisherWithID(reqCopy.Site.Publisher, publisherID) + siteCopy := *request.Site + siteCopy.Publisher = &publisher + reqCopy.Site = &siteCopy + } + } + + encoded, err := json.Marshal(reqCopy) + if err != nil { + errs = append(errs, err) + return nil, errs + } + + headers := http.Header{} + headers.Add("Content-Type", "application/json;charset=utf-8") + + reqBidder := &adapters.RequestData{ + Method: "POST", + Uri: adapter.endpoint, + Body: encoded, + Headers: headers, + } + reqsBidder = append(reqsBidder, reqBidder) + return +} + +type amxBidExt struct { + Himp []string `json:"himp,omitempty"` + StartDelay *int `json:"startdelay,omitempty"` +} + +// MakeBids will parse the bids from the AMX server +func (adapter *AMXAdapter) MakeBids(request *openrtb.BidRequest, externalRequest *adapters.RequestData, response *adapters.ResponseData) (*adapters.BidderResponse, []error) { + var errs []error + + if http.StatusNoContent == response.StatusCode { + return nil, nil + } + + if http.StatusBadRequest == response.StatusCode { + internalNBR := response.Headers.Get(nbrHeaderName) + return nil, []error{&errortypes.BadInput{ + Message: fmt.Sprintf("Invalid Request: 400. Error Code: %s", internalNBR), + }} + } + + if http.StatusOK != response.StatusCode { + internalNBR := response.Headers.Get(nbrHeaderName) + return nil, []error{&errortypes.BadServerResponse{ + Message: fmt.Sprintf("Unexpected response: %d. Error Code: %s", response.StatusCode, internalNBR), + }} + } + + var bidResp openrtb.BidResponse + if err := json.Unmarshal(response.Body, &bidResp); err != nil { + return nil, []error{err} + } + + bidResponse := adapters.NewBidderResponseWithBidsCapacity(5) + + for _, sb := range bidResp.SeatBid { + for _, bid := range sb.Bid { + bidExt, bidExtErr := getBidExt(bid.Ext) + if bidExtErr != nil { + errs = append(errs, bidExtErr) + continue + } + + bidType := getMediaTypeForBid(bidExt) + b := &adapters.TypedBid{ + Bid: &bid, + BidType: bidType, + } + if b.BidType == openrtb_ext.BidTypeVideo { + b.Bid.AdM = interpolateImpressions(bid, bidExt) + // remove the NURL so a client/player doesn't fire it twice + b.Bid.NURL = "" + } + bidResponse.Bids = append(bidResponse.Bids, b) + } + } + return bidResponse, errs + +} + +func getBidExt(ext json.RawMessage) (amxBidExt, error) { + if len(ext) == 0 { + return amxBidExt{}, nil + } + + var bidExt amxBidExt + err := json.Unmarshal(ext, &bidExt) + return bidExt, err +} + +func getMediaTypeForBid(bidExt amxBidExt) openrtb_ext.BidType { + if bidExt.StartDelay != nil { + return openrtb_ext.BidTypeVideo + } + + return openrtb_ext.BidTypeBanner +} + +func pixelToImpression(pixel string) string { + return fmt.Sprintf(vastImpressionFormat, pixel) +} + +func interpolateImpressions(bid openrtb.Bid, ext amxBidExt) string { + var buffer strings.Builder + if bid.NURL != "" { + buffer.WriteString(pixelToImpression(bid.NURL)) + } + + for _, impPixel := range ext.Himp { + if impPixel != "" { + buffer.WriteString(pixelToImpression(impPixel)) + } + } + + results := strings.Replace(bid.AdM, vastSearchPoint, vastSearchPoint+buffer.String(), 1) + return results +} diff --git a/adapters/amx/amx_test.go b/adapters/amx/amx_test.go new file mode 100644 index 00000000000..3d6e118772f --- /dev/null +++ b/adapters/amx/amx_test.go @@ -0,0 +1,180 @@ +package amx + +import ( + "encoding/json" + "fmt" + "regexp" + "testing" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/stretchr/testify/assert" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adapterstest" +) + +const ( + amxTestEndpoint = "http://pbs-dev.amxrtb.com/auction/openrtb" + sampleVastADM = "00:00:15" + sampleDisplayADM = "" +) + +func TestJsonSamples(t *testing.T) { + adapterstest.RunJSONBidderTest(t, "amxtest", NewAMXBidder(amxTestEndpoint)) +} + +func TestMakeRequestsTagID(t *testing.T) { + var w, h int = 300, 250 + var width, height uint64 = uint64(w), uint64(h) + adapter := NewAMXBidder(amxTestEndpoint) + + type testCase struct { + tagID string + extAdUnitID string + expectedTagID string + blankNil bool + } + + tests := []testCase{ + {tagID: "tag-id", extAdUnitID: "ext.adUnitID", expectedTagID: "ext.adUnitID", blankNil: false}, + {tagID: "tag-id", extAdUnitID: "", expectedTagID: "tag-id", blankNil: false}, + {tagID: "tag-id", extAdUnitID: "", expectedTagID: "tag-id", blankNil: true}, + {tagID: "", extAdUnitID: "", expectedTagID: "", blankNil: true}, + {tagID: "", extAdUnitID: "", expectedTagID: "", blankNil: false}, + {tagID: "", extAdUnitID: "ext.adUnitID", expectedTagID: "ext.adUnitID", blankNil: true}, + {tagID: "", extAdUnitID: "ext.adUnitID", expectedTagID: "ext.adUnitID", blankNil: false}, + } + + for _, tc := range tests { + imp1 := openrtb.Imp{ + ID: "sample_imp_1", + Banner: &openrtb.Banner{ + W: &width, + H: &height, + Format: []openrtb.Format{ + {W: 300, H: 250}, + }, + }} + + if tc.extAdUnitID != "" || !tc.blankNil { + imp1.Ext = json.RawMessage( + fmt.Sprintf(`{"bidder":{"adUnitId":"%s"}}`, tc.extAdUnitID)) + } + + if tc.tagID != "" || !tc.blankNil { + imp1.TagID = tc.tagID + } + + inputRequest := openrtb.BidRequest{ + User: &openrtb.User{}, + Imp: []openrtb.Imp{imp1}, + Site: &openrtb.Site{}, + } + + actualAdapterRequests, err := adapter.MakeRequests(&inputRequest, &adapters.ExtraRequestInfo{}) + assert.Len(t, actualAdapterRequests, 1) + assert.Empty(t, err) + var body openrtb.BidRequest + assert.Nil(t, json.Unmarshal(actualAdapterRequests[0].Body, &body)) + assert.Equal(t, tc.expectedTagID, body.Imp[0].TagID) + } +} + +func TestMakeRequestsPublisherId(t *testing.T) { + var w, h int = 300, 250 + var width, height uint64 = uint64(w), uint64(h) + adapter := NewAMXBidder(amxTestEndpoint) + + type testCase struct { + publisherID string + extTagID string + expectedPublisherID string + blankNil bool + } + + tests := []testCase{ + {publisherID: "publisher.id", extTagID: "ext.tagId", expectedPublisherID: "ext.tagId", blankNil: false}, + {publisherID: "publisher.id", extTagID: "", expectedPublisherID: "publisher.id", blankNil: false}, + {publisherID: "", extTagID: "ext.tagId", expectedPublisherID: "ext.tagId", blankNil: false}, + {publisherID: "", extTagID: "ext.tagId", expectedPublisherID: "ext.tagId", blankNil: true}, + {publisherID: "publisher.id", extTagID: "", expectedPublisherID: "publisher.id", blankNil: false}, + {publisherID: "publisher.id", extTagID: "", expectedPublisherID: "publisher.id", blankNil: true}, + } + + for _, tc := range tests { + imp1 := openrtb.Imp{ + ID: "sample_imp_1", + Banner: &openrtb.Banner{ + W: &width, + H: &height, + Format: []openrtb.Format{ + {W: 300, H: 250}, + }, + }} + + if tc.extTagID != "" || !tc.blankNil { + imp1.Ext = json.RawMessage( + fmt.Sprintf(`{"bidder":{"tagId":"%s"}}`, tc.extTagID)) + } + + inputRequest := openrtb.BidRequest{ + User: &openrtb.User{ID: "example_user_id"}, + Imp: []openrtb.Imp{imp1}, + Site: &openrtb.Site{}, + ID: "1234", + } + + if tc.publisherID != "" || !tc.blankNil { + inputRequest.Site.Publisher = &openrtb.Publisher{ + ID: tc.publisherID, + } + } + + actualAdapterRequests, err := adapter.MakeRequests(&inputRequest, &adapters.ExtraRequestInfo{}) + assert.Len(t, actualAdapterRequests, 1) + assert.Empty(t, err) + var body openrtb.BidRequest + assert.Nil(t, json.Unmarshal(actualAdapterRequests[0].Body, &body)) + assert.Equal(t, tc.expectedPublisherID, body.Site.Publisher.ID) + } +} + +var vastImpressionRXP = regexp.MustCompile(``) + +func countImpressionPixels(vast string) int { + matches := vastImpressionRXP.FindAllIndex([]byte(vast), -1) + return len(matches) +} + +func TestVideoImpInsertion(t *testing.T) { + markup := interpolateImpressions(openrtb.Bid{ + AdM: sampleVastADM, + NURL: "https://example2.com/nurl", + }, amxBidExt{Himp: []string{"https://example.com/pixel.png"}}) + assert.Contains(t, markup, "example2.com/nurl") + assert.Contains(t, markup, "example.com/pixel.png") + assert.Equal(t, 3, countImpressionPixels(markup), "should have 3 Impression pixels") + + // make sure that a blank NURL won't result in a blank impression tag + markup = interpolateImpressions(openrtb.Bid{ + AdM: sampleVastADM, + NURL: "", + }, amxBidExt{}) + assert.Equal(t, 1, countImpressionPixels(markup), "should have 1 impression pixels") + + // we should also ignore blank ext.Himp pixels + markup = interpolateImpressions(openrtb.Bid{ + AdM: sampleVastADM, + NURL: "https://example-nurl.com/nurl", + }, amxBidExt{Himp: []string{"", "", ""}}) + assert.Equal(t, 2, countImpressionPixels(markup), "should have 2 impression pixels") +} + +func TestNoDisplayImpInsertion(t *testing.T) { + data := interpolateImpressions(openrtb.Bid{ + AdM: sampleDisplayADM, + NURL: "https://example2.com/nurl", + }, amxBidExt{Himp: []string{"https://example.com/pixel.png"}}) + assert.NotContains(t, data, "example2.com/nurl") + assert.NotContains(t, data, "example.com/pixel.png") +} diff --git a/adapters/amx/amxtest/exemplary/app-simple.json b/adapters/amx/amxtest/exemplary/app-simple.json new file mode 100644 index 00000000000..b2f538493da --- /dev/null +++ b/adapters/amx/amxtest/exemplary/app-simple.json @@ -0,0 +1,178 @@ +{ + "mockBidRequest": { + "allimps": 0, + "app": { + "bundle": "639881495", + "name": "Imgur (iOS)", + "storeurl": "https://apps.apple.com/us/app/imgur-meme-gif-maker/id639881495", + "ver": "2020.4", + "publisher": { + "id": "unused-overridden" + } + }, + "device": { + "connectiontype": 2, + "dnt": 0, + "h": 1024, + "ifa": "201461F8-0F14-4ADD-A87F-AAAAAAAAA", + "ip": "73.221.0.0", + "language": "en", + "os": "ios", + "ua": "Mozilla/5.0 (iPad; CPU OS 12_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148", + "w": 768 + }, + "id": "WQ5V2DWVTMNXRSWL", + "imp": [ + { + "banner": { + "api": [ + 3, + 5 + ], + "format": [ + { + "h": 50, + "w": 320 + } + ], + "h": 50, + "pos": 1, + "topframe": 1, + "w": 320 + }, + "bidfloor": 0.5, + "bidfloorcur": "USD", + "clickbrowser": 1, + "id": "1", + "instl": 1, + "secure": 1, + "ext": { + "bidder": { + "tagId": "cHJlYmlkLm9yZw", + "adUnitId": "sample-ad-unit-id" + } + } + } + ], + "regs": { + "ext": { + "gdpr": 0 + } + }, + "tmax": 300 + }, + "httpCalls": [ + { + "expectedRequest": { + "uri": "http://pbs-dev.amxrtb.com/auction/openrtb?v=pbs1.0", + "body": { + "app": { + "bundle": "639881495", + "name": "Imgur (iOS)", + "storeurl": "https://apps.apple.com/us/app/imgur-meme-gif-maker/id639881495", + "ver": "2020.4", + "publisher": { + "id": "cHJlYmlkLm9yZw" + } + }, + "device": { + "connectiontype": 2, + "dnt": 0, + "h": 1024, + "ifa": "201461F8-0F14-4ADD-A87F-AAAAAAAAA", + "ip": "73.221.0.0", + "language": "en", + "os": "ios", + "ua": "Mozilla/5.0 (iPad; CPU OS 12_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148", + "w": 768 + }, + "id": "WQ5V2DWVTMNXRSWL", + "imp": [ + { + "banner": { + "api": [ + 3, + 5 + ], + "format": [ + { + "h": 50, + "w": 320 + } + ], + "h": 50, + "pos": 1, + "topframe": 1, + "w": 320 + }, + "bidfloor": 0.5, + "bidfloorcur": "USD", + "clickbrowser": 1, + "id": "1", + "instl": 1, + "secure": 1, + "tagid": "sample-ad-unit-id", + "ext": { + "bidder": { + "tagId": "cHJlYmlkLm9yZw", + "adUnitId": "sample-ad-unit-id" + } + } + } + ], + "regs": { + "ext": { + "gdpr": 0 + } + }, + "tmax": 300 + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "WQ5V2DWVTMNXRSWL", + "seatbid": [{ + "bid": [{ + "id": "TEST", + "impid": "1", + "price": 10.0, + "adid": "1", + "adm": "", + "adomain": ["amxrtb.com"], + "iurl": "https://assets.a-mo.net/300x250.v2.png", + "cid": "1", + "crid": "1", + "h": 300, + "w": 250 + }] + }], + "cur": "USD" + } + } + } + ], + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "TEST", + "impid": "1", + "price": 10.0, + "adid": "1", + "adm": "", + "adomain": ["amxrtb.com"], + "iurl": "https://assets.a-mo.net/300x250.v2.png", + "cid": "1", + "crid": "1", + "h": 300, + "w": 250 + }, + "type": "banner" + } + ] + } + ] +} diff --git a/adapters/amx/amxtest/exemplary/video-simple.json b/adapters/amx/amxtest/exemplary/video-simple.json new file mode 100644 index 00000000000..8fb3baa26d0 --- /dev/null +++ b/adapters/amx/amxtest/exemplary/video-simple.json @@ -0,0 +1,245 @@ +{ + "mockBidRequest": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "boxingallowed": 1, + "linearity": 1, + "maxduration": 90, + "minduration": 6, + "mimes": ["video/mp4"], + "placement": 1, + "playbackmethod": [2], + "protocols": [1,2,3,4,5,6,7,8], + "skip": 1, + "skipafter": 5, + "startdelay": 0, + "h": 300, + "pos": 1, + "w": 640 + }, + "ext": { + "bidder": { + "tagId": "cHJlYmlkLm9yZw", + "adUnitId": "tagid-override" + } + }, + "id": "1", + "secure": 1 + } + ], + "regs": { + "ext": { + "gdpr": 0, + "us_privacy": "1---" + } + }, + "site": { + "domain": "www.example.com", + "ext": { + "amp": 0 + }, + "publisher": { + "id": "unused_publisher_id" + }, + "page": "https://www.example.com/es6/es6_objects.htm", + "ref": "https://www.example.com/es6/es6_objects.htm" + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "test": 0, + "tmax": 300, + "user": { + "ext": { + "eids": [ + { + "source": "amxid", + "uids": [ + { + "atype": 1, + "id": "88de601e-3d98-48e7-81d7-00000000" + } + ] + } + ], + "gdpr": 0, + "us_privacy": "1---" + } + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pbs-dev.amxrtb.com/auction/openrtb?v=pbs1.0", + "body": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "boxingallowed": 1, + "linearity": 1, + "maxduration": 90, + "minduration": 6, + "mimes": ["video/mp4"], + "placement": 1, + "playbackmethod": [2], + "protocols": [1,2,3,4,5,6,7,8], + "skip": 1, + "skipafter": 5, + "startdelay": 0, + "h": 300, + "pos": 1, + "w": 640 + }, + "ext": { + "bidder": { + "tagId": "cHJlYmlkLm9yZw", + "adUnitId": "tagid-override" + } + }, + "tagid": "tagid-override", + "id": "1", + "secure": 1 + } + ], + "regs": { + "ext": { + "gdpr": 0, + "us_privacy": "1---" + } + }, + "site": { + "domain": "www.example.com", + "ext": { + "amp": 0 + }, + "publisher": { + "id": "cHJlYmlkLm9yZw" + }, + "page": "https://www.example.com/es6/es6_objects.htm", + "ref": "https://www.example.com/es6/es6_objects.htm" + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "tmax": 300, + "user": { + "ext": { + "eids": [ + { + "source": "amxid", + "uids": [ + { + "atype": 1, + "id": "88de601e-3d98-48e7-81d7-00000000" + } + ] + } + ], + "gdpr": 0, + "us_privacy": "1---" + } + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "WQ5V2DWVTMNXABDD", + "seatbid": [{ + "bid": [{ + "id": "TEST", + "impid": "1", + "price": 10.0, + "adid": "1", + "adm": "00:00:15", + "nurl": "https://example.com/nurl", + "adomain": ["amxrtb.com"], + "iurl": "https://assets.a-mo.net/300x250.v2.png", + "cid": "1", + "crid": "1", + "h": 600, + "w": 300, + "ext": { + "himp": ["https://example.com/imp-tracker/pixel.gif?param=1¶m2=2"], + "startdelay": 0 + } + }] + }], + "cur": "USD" + } + } + }], + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "TEST", + "impid": "1", + "price": 10.0, + "adid": "1", + "adm": "00:00:15", + "adomain": ["amxrtb.com"], + "iurl": "https://assets.a-mo.net/300x250.v2.png", + "cid": "1", + "crid": "1", + "ext": { + "himp": ["https://example.com/imp-tracker/pixel.gif?param=1¶m2=2"], + "startdelay": 0 + }, + "h": 600, + "w": 300 + }, + "type": "video" + } + ] + } + ] +} \ No newline at end of file diff --git a/adapters/amx/amxtest/exemplary/web-simple.json b/adapters/amx/amxtest/exemplary/web-simple.json new file mode 100644 index 00000000000..74854f912ae --- /dev/null +++ b/adapters/amx/amxtest/exemplary/web-simple.json @@ -0,0 +1,246 @@ +{ + "mockBidRequest": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "banner": { + "format": [ + { + "h": 600, + "w": 300 + } + ], + "h": 600, + "pos": 1, + "w": 300 + }, + "ext": { + "bidder": { + "tagId": "cHJlYmlkLm9yZw" + } + }, + "tagid": "example-tag-id", + "id": "1", + "secure": 1 + } + ], + "regs": { + "ext": { + "gdpr": 0, + "us_privacy": "1---" + } + }, + "site": { + "domain": "www.example.com", + "ext": { + "amp": 0 + }, + "publisher": { + "id": "unused_publisher_id" + }, + "page": "https://www.example.com/es6/es6_objects.htm", + "ref": "https://www.example.com/es6/es6_objects.htm" + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "test": 0, + "tmax": 300, + "user": { + "ext": { + "eids": [ + { + "source": "amxid", + "uids": [ + { + "atype": 1, + "id": "88de601e-3d98-48e7-81d7-00000000" + } + ] + }, + { + "source": "adserver.org", + "uids": [ + { + "id": "1234567", + "ext": { + "rtiPartner": "TDID" + } + } + ] + } + ], + "gdpr": 0, + "us_privacy": "1---" + } + } + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pbs-dev.amxrtb.com/auction/openrtb?v=pbs1.0", + "body": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "tagid": "example-tag-id", + "banner": { + "format": [ + { + "h": 600, + "w": 300 + } + ], + "h": 600, + "pos": 1, + "w": 300 + }, + "ext": { + "bidder": { + "tagId": "cHJlYmlkLm9yZw" + } + }, + "id": "1", + "secure": 1 + } + ], + "regs": { + "ext": { + "gdpr": 0, + "us_privacy": "1---" + } + }, + "site": { + "domain": "www.example.com", + "ext": { + "amp": 0 + }, + "publisher": { + "id": "cHJlYmlkLm9yZw" + }, + "page": "https://www.example.com/es6/es6_objects.htm", + "ref": "https://www.example.com/es6/es6_objects.htm" + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "tmax": 300, + "user": { + "ext": { + "eids": [ + { + "source": "amxid", + "uids": [ + { + "atype": 1, + "id": "88de601e-3d98-48e7-81d7-00000000" + } + ] + }, + { + "source": "adserver.org", + "uids": [ + { + "id": "1234567", + "ext": { + "rtiPartner": "TDID" + } + } + ] + } + ], + "gdpr": 0, + "us_privacy": "1---" + } + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "WQ5V2DWVTMNXABDD", + "seatbid": [{ + "bid": [{ + "id": "TEST", + "impid": "1", + "price": 10.0, + "adid": "1", + "adm": "", + "adomain": ["amxrtb.com"], + "iurl": "https://assets.a-mo.net/300x250.v2.png", + "cid": "1", + "crid": "1", + "h": 600, + "w": 300 + }] + }], + "cur": "USD" + } + } + }], + + "expectedBidResponses": [ + { + "currency": "USD", + "bids": [ + { + "bid": { + "id": "TEST", + "impid": "1", + "price": 10.0, + "adid": "1", + "adm": "", + "adomain": ["amxrtb.com"], + "iurl": "https://assets.a-mo.net/300x250.v2.png", + "cid": "1", + "crid": "1", + "h": 600, + "w": 300 + }, + "type": "banner" + } + ] + } + ] +} \ No newline at end of file diff --git a/adapters/amx/amxtest/params/race/display.json b/adapters/amx/amxtest/params/race/display.json new file mode 100644 index 00000000000..bd101e95a25 --- /dev/null +++ b/adapters/amx/amxtest/params/race/display.json @@ -0,0 +1 @@ +{"tagId":"sample345", "adUnitId": "sampleAdUnitID"} \ No newline at end of file diff --git a/adapters/amx/amxtest/params/race/video.json b/adapters/amx/amxtest/params/race/video.json new file mode 100644 index 00000000000..d2f11bf80b4 --- /dev/null +++ b/adapters/amx/amxtest/params/race/video.json @@ -0,0 +1 @@ +{"tagId": "sample123", "adUnitId": "sampleAdUnitID"} \ No newline at end of file diff --git a/adapters/amx/amxtest/supplemental/204-response.json b/adapters/amx/amxtest/supplemental/204-response.json new file mode 100644 index 00000000000..09571a03569 --- /dev/null +++ b/adapters/amx/amxtest/supplemental/204-response.json @@ -0,0 +1,109 @@ +{ + "mockBidRequest": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "h": 300, + "pos": 1, + "w": 640 + }, + "id": "1", + "secure": 1 + } + ], + "site": { + "ext": { + "amp": 0 + } + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "test": 0, + "tmax": 300 + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pbs-dev.amxrtb.com/auction/openrtb?v=pbs1.0", + "body": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "mimes": null, + "h": 300, + "pos": 1, + "w": 640 + }, + "id": "1", + "secure": 1 + } + ], + "site": { + "ext": { + "amp": 0 + } + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "tmax": 300 + } + }, + "mockResponse": { + "status": 204, + "headers": { + "X-Nbr": [ + "3b" + ] + }, + "body": {} + } + }], + "expectedMakeBidsErrors": [] +} diff --git a/adapters/amx/amxtest/supplemental/400-response.json b/adapters/amx/amxtest/supplemental/400-response.json new file mode 100644 index 00000000000..f10cea89718 --- /dev/null +++ b/adapters/amx/amxtest/supplemental/400-response.json @@ -0,0 +1,114 @@ +{ + "mockBidRequest": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "h": 300, + "pos": 1, + "w": 640 + }, + "id": "1", + "secure": 1 + } + ], + "site": { + "ext": { + "amp": 0 + } + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "test": 0, + "tmax": 300 + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pbs-dev.amxrtb.com/auction/openrtb?v=pbs1.0", + "body": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "mimes": null, + "h": 300, + "pos": 1, + "w": 640 + }, + "id": "1", + "secure": 1 + } + ], + "site": { + "ext": { + "amp": 0 + } + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "tmax": 300 + } + }, + "mockResponse": { + "status": 400, + "headers": { + "X-Nbr": [ + "3b" + ] + }, + "body": {} + } + }], + "expectedMakeBidsErrors": [ + { + "value": "Invalid Request: 400. Error Code: 3b", + "comparison": "literal" + } + ] +} diff --git a/adapters/amx/amxtest/supplemental/500-response.json b/adapters/amx/amxtest/supplemental/500-response.json new file mode 100644 index 00000000000..fe5d89930c8 --- /dev/null +++ b/adapters/amx/amxtest/supplemental/500-response.json @@ -0,0 +1,114 @@ +{ + "mockBidRequest": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "h": 300, + "pos": 1, + "w": 640 + }, + "id": "1", + "secure": 1 + } + ], + "site": { + "ext": { + "amp": 0 + } + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "test": 0, + "tmax": 300 + }, + "httpCalls": [{ + "expectedRequest": { + "uri": "http://pbs-dev.amxrtb.com/auction/openrtb?v=pbs1.0", + "body": { + "device": { + "dnt": 0, + "h": 1120, + "ip": "98.249.0.0", + "language": "en", + "os": "macos", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", + "w": 1792 + }, + "id": "TL3JS6F43CKNDQFY", + "imp": [ + { + "video": { + "api": [1,2], + "mimes": null, + "h": 300, + "pos": 1, + "w": 640 + }, + "id": "1", + "secure": 1 + } + ], + "site": { + "ext": { + "amp": 0 + } + }, + "source": { + "ext": { + "schain": { + "complete": 1, + "nodes": [ + { + "asi": "amxrtb.com", + "hp": 1, + "sid": "1234" + } + ], + "ver": "1.0" + } + } + }, + "tmax": 300 + } + }, + "mockResponse": { + "status": 500, + "headers": { + "X-Nbr": [ + "7a" + ] + }, + "body": {} + } + }], + "expectedMakeBidsErrors": [ + { + "value": "Unexpected response: 500. Error Code: 7a", + "comparison": "literal" + } + ] +} diff --git a/adapters/amx/params_test.go b/adapters/amx/params_test.go new file mode 100644 index 00000000000..ef177644b21 --- /dev/null +++ b/adapters/amx/params_test.go @@ -0,0 +1,47 @@ +package amx + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/stretchr/testify/assert" +) + +var validBidParams = []string{ + `{"tagId":"sampleTagId", "adUnitId": "sampleAdUnitId"}`, + `{"tagId":"sampleTagId", "adUnitId": ""}`, + `{"adUnitId": ""}`, + `{"adUnitId": "sampleAdUnitId"}`, + `{"tagId":"sampleTagId"}`, + `{"tagId":""}`, + `{}`, + `{"otherValue": "ignored"}`, + `{"tagId": "sampleTagId", "otherValue": "ignored"}`, + `{"otherValue": "ignored", "adUnitId": "sampleAdUnitId"}`, +} + +var invalidBidParams = []string{ + `{"tagId":1234}`, + `{"tagId": true}`, + `{"adUnitId": true}`, + `{"adUnitId": null}`, + `{"adUnitId": null, "tagId": "sampleTagId"}`, + `{"adUnitId": 1234, "tagId": "sampleTagId"}`, +} + +func TestValidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + assert.Nil(t, err) + for _, params := range validBidParams { + assert.Nil(t, validator.Validate(openrtb_ext.BidderAMX, json.RawMessage(params))) + } +} + +func TestInValidParams(t *testing.T) { + validator, err := openrtb_ext.NewBidderParamsValidator("../../static/bidder-params") + assert.Nil(t, err) + for _, params := range invalidBidParams { + assert.NotNil(t, validator.Validate(openrtb_ext.BidderAMX, json.RawMessage(params))) + } +} diff --git a/adapters/amx/usersync.go b/adapters/amx/usersync.go new file mode 100644 index 00000000000..d9ff10df562 --- /dev/null +++ b/adapters/amx/usersync.go @@ -0,0 +1,13 @@ +package amx + +import ( + "text/template" + + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/usersync" +) + +// NewAMXSyncer produces an AMX RTB usersyncer +func NewAMXSyncer(temp *template.Template) usersync.Usersyncer { + return adapters.NewSyncer("amx", 737, temp, adapters.SyncTypeRedirect) +} diff --git a/adapters/amx/usersync_test.go b/adapters/amx/usersync_test.go new file mode 100644 index 00000000000..e6020b27570 --- /dev/null +++ b/adapters/amx/usersync_test.go @@ -0,0 +1,23 @@ +package amx + +import ( + "testing" + "text/template" + + "github.com/PubMatic-OpenWrap/prebid-server/privacy" + "github.com/stretchr/testify/assert" +) + +func TestAMXSyncer(t *testing.T) { + syncURL := "http://pbs.amxrtb.com/cchain/0?gdpr={{.GDPR}}&gdpr_consent={{.GDPRConsent}}&cb=localhost%2Fsetuid%3Fbidder%3Damx%26uid%3D" + syncURLTemplate := template.Must(template.New("sync-template").Parse(syncURL)) + + syncer := NewAMXSyncer(syncURLTemplate) + syncInfo, err := syncer.GetUsersyncInfo(privacy.Policies{}) + + assert.NoError(t, err) + assert.Equal(t, "http://pbs.amxrtb.com/cchain/0?gdpr=&gdpr_consent=&cb=localhost%2Fsetuid%3Fbidder%3Damx%26uid%3D", syncInfo.URL) + assert.Equal(t, "redirect", syncInfo.Type) + assert.EqualValues(t, 737, syncer.GDPRVendorID()) + assert.Equal(t, false, syncInfo.SupportCORS) +} diff --git a/adapters/appnexus/appnexus.go b/adapters/appnexus/appnexus.go index 1b3b42295d7..145c830dbb6 100644 --- a/adapters/appnexus/appnexus.go +++ b/adapters/appnexus/appnexus.go @@ -6,6 +6,7 @@ import ( "encoding/json" "fmt" "io/ioutil" + "math/rand" "net/http" "strconv" "strings" @@ -95,10 +96,11 @@ type appnexusBidExt struct { } type appnexusReqExtAppnexus struct { - IncludeBrandCategory *bool `json:"include_brand_category,omitempty"` - BrandCategoryUniqueness *bool `json:"brand_category_uniqueness,omitempty"` - IsAMP int `json:"is_amp,omitempty"` - HeaderBiddingSource int `json:"hb_source,omitempty"` + IncludeBrandCategory *bool `json:"include_brand_category,omitempty"` + BrandCategoryUniqueness *bool `json:"brand_category_uniqueness,omitempty"` + IsAMP int `json:"is_amp,omitempty"` + HeaderBiddingSource int `json:"hb_source,omitempty"` + AdPodId string `json:"adpod_id,omitempty"` } // Full request extension including appnexus extension object @@ -354,14 +356,56 @@ func (a *AppNexusAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *ada } reqExt.Appnexus.IsAMP = isAMP reqExt.Appnexus.HeaderBiddingSource = a.hbSource + isVIDEO + + imps := request.Imp + + // For long form requests adpod_id must be sent downstream. + // Adpod id is a unique identifier for pod + // All impressions in the same pod must have the same pod id in request extension + // For this all impressions in request should belong to the same pod + // If impressions number per pod is more than maxImpsPerReq - divide those imps to several requests but keep pod id the same + if isVIDEO == 1 { + podImps := groupByPods(imps) + + requests := make([]*adapters.RequestData, 0, len(podImps)) + for _, podImps := range podImps { + reqExt.Appnexus.AdPodId = generatePodId() + + reqs, errors := splitRequests(podImps, request, reqExt, thisURI, errs) + requests = append(requests, reqs...) + errs = append(errs, errors...) + } + return requests, errs + } + + return splitRequests(imps, request, reqExt, thisURI, errs) +} + +func generatePodId() string { + val := rand.Int63() + return fmt.Sprint(val) +} + +func groupByPods(imps []openrtb.Imp) map[string]([]openrtb.Imp) { + // find number of pods in response + podImps := make(map[string][]openrtb.Imp) + for _, imp := range imps { + pod := strings.Split(imp.ID, "_")[0] + podImps[pod] = append(podImps[pod], imp) + } + return podImps +} + +func marshalAndSetRequestExt(request *openrtb.BidRequest, requestExtension appnexusReqExt, errs []error) { var err error - request.Ext, err = json.Marshal(reqExt) + request.Ext, err = json.Marshal(requestExtension) if err != nil { errs = append(errs, err) - return nil, errs } +} + +func splitRequests(imps []openrtb.Imp, request *openrtb.BidRequest, requestExtension appnexusReqExt, uri string, errs []error) ([]*adapters.RequestData, []error) { - imps := request.Imp // Initial capacity for future array of requests, memory optimization. // Let's say there are 35 impressions and limit impressions per request equals to 10. // In this case we need to create 4 requests with 10, 10, 10 and 5 impressions. @@ -375,6 +419,8 @@ func (a *AppNexusAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *ada headers.Add("Content-Type", "application/json;charset=utf-8") headers.Add("Accept", "application/json") + marshalAndSetRequestExt(request, requestExtension, errs) + for impsLeft { endInd := startInd + maxImpsPerReq @@ -393,7 +439,7 @@ func (a *AppNexusAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo *ada resArr = append(resArr, &adapters.RequestData{ Method: "POST", - Uri: thisURI, + Uri: uri, Body: reqJSON, Headers: headers, }) diff --git a/adapters/appnexus/appnexus_test.go b/adapters/appnexus/appnexus_test.go index c6f537996b9..7468250b28d 100644 --- a/adapters/appnexus/appnexus_test.go +++ b/adapters/appnexus/appnexus_test.go @@ -4,9 +4,11 @@ import ( "bytes" "context" "encoding/json" + "github.com/stretchr/testify/assert" "io/ioutil" "net/http" "net/http/httptest" + "regexp" "testing" "time" @@ -38,6 +40,233 @@ func TestMemberQueryParam(t *testing.T) { } } +func TestVideoSinglePod(t *testing.T) { + var a AppNexusAdapter + a.URI = "http://test.com/openrtb2" + a.hbSource = 5 + + var reqInfo adapters.ExtraRequestInfo + reqInfo.PbsEntryPoint = "video" + + var req openrtb.BidRequest + req.ID = "test_id" + + reqExt := `{"prebid":{}}` + impExt := `{"bidder":{"placementId":123}}` + req.Ext = []byte(reqExt) + + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_0", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_1", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_2", Ext: []byte(impExt)}) + + result, err := a.MakeRequests(&req, &reqInfo) + + assert.Empty(t, err, "Errors array should be empty") + assert.Len(t, result, 1, "Only one request should be returned") + + var error error + var reqData *openrtb.BidRequest + error = json.Unmarshal(result[0].Body, &reqData) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt *appnexusReqExt + error = json.Unmarshal(reqData.Ext, &reqDataExt) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + regMatch, matchErr := regexp.Match(`[0-9]19`, []byte(reqDataExt.Appnexus.AdPodId)) + assert.NoError(t, matchErr, "Regex match error should be nil") + assert.True(t, regMatch, "AdPod id doesn't present in Appnexus extension or has incorrect format") +} + +func TestVideoSinglePodManyImps(t *testing.T) { + var a AppNexusAdapter + a.URI = "http://test.com/openrtb2" + a.hbSource = 5 + + var reqInfo adapters.ExtraRequestInfo + reqInfo.PbsEntryPoint = "video" + + var req openrtb.BidRequest + req.ID = "test_id" + + reqExt := `{"prebid":{}}` + impExt := `{"bidder":{"placementId":123}}` + req.Ext = []byte(reqExt) + + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_0", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_1", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_2", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_3", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_4", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_5", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_6", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_7", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_8", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_9", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_10", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_11", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_12", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_13", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_14", Ext: []byte(impExt)}) + + res, err := a.MakeRequests(&req, &reqInfo) + + assert.Empty(t, err, "Errors array should be empty") + assert.Len(t, res, 2, "Two requests should be returned") + + var error error + var reqData1 *openrtb.BidRequest + error = json.Unmarshal(res[0].Body, &reqData1) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt1 *appnexusReqExt + error = json.Unmarshal(reqData1.Ext, &reqDataExt1) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + adPodId1 := reqDataExt1.Appnexus.AdPodId + + var reqData2 *openrtb.BidRequest + error = json.Unmarshal(res[1].Body, &reqData2) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt2 *appnexusReqExt + error = json.Unmarshal(reqData2.Ext, &reqDataExt2) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + adPodId2 := reqDataExt2.Appnexus.AdPodId + + assert.Equal(t, adPodId1, adPodId2, "AdPod id is not the same for the same pod") +} + +func TestVideoTwoPods(t *testing.T) { + var a AppNexusAdapter + a.URI = "http://test.com/openrtb2" + a.hbSource = 5 + + var reqInfo adapters.ExtraRequestInfo + reqInfo.PbsEntryPoint = "video" + + var req openrtb.BidRequest + req.ID = "test_id" + + reqExt := `{"prebid":{}}` + impExt := `{"bidder":{"placementId":123}}` + req.Ext = []byte(reqExt) + + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_0", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_1", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_2", Ext: []byte(impExt)}) + + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_0", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_1", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_2", Ext: []byte(impExt)}) + + res, err := a.MakeRequests(&req, &reqInfo) + + assert.Empty(t, err, "Errors array should be empty") + assert.Len(t, res, 2, "Two request should be returned") + + var error error + var reqData1 *openrtb.BidRequest + error = json.Unmarshal(res[0].Body, &reqData1) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt1 *appnexusReqExt + error = json.Unmarshal(reqData1.Ext, &reqDataExt1) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + adPodId1 := reqDataExt1.Appnexus.AdPodId + + var reqData2 *openrtb.BidRequest + error = json.Unmarshal(res[1].Body, &reqData2) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt2 *appnexusReqExt + error = json.Unmarshal(reqData2.Ext, &reqDataExt2) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + adPodId2 := reqDataExt2.Appnexus.AdPodId + + assert.NotEqual(t, adPodId1, adPodId2, "AdPod id should be different for different pods") +} + +func TestVideoTwoPodsManyImps(t *testing.T) { + var a AppNexusAdapter + a.URI = "http://test.com/openrtb2" + a.hbSource = 5 + + var reqInfo adapters.ExtraRequestInfo + reqInfo.PbsEntryPoint = "video" + + var req openrtb.BidRequest + req.ID = "test_id" + + reqExt := `{"prebid":{}}` + impExt := `{"bidder":{"placementId":123}}` + req.Ext = []byte(reqExt) + + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_0", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_1", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "1_2", Ext: []byte(impExt)}) + + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_0", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_1", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_2", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_3", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_4", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_5", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_6", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_7", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_8", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_9", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_10", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_11", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_12", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_13", Ext: []byte(impExt)}) + req.Imp = append(req.Imp, openrtb.Imp{ID: "2_14", Ext: []byte(impExt)}) + + res, err := a.MakeRequests(&req, &reqInfo) + + assert.Empty(t, err, "Errors array should be empty") + assert.Len(t, res, 3, "Three requests should be returned") + + var error error + var reqData1 *openrtb.BidRequest + error = json.Unmarshal(res[0].Body, &reqData1) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt1 *appnexusReqExt + error = json.Unmarshal(reqData1.Ext, &reqDataExt1) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + var reqData2 *openrtb.BidRequest + error = json.Unmarshal(res[1].Body, &reqData2) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt2 *appnexusReqExt + error = json.Unmarshal(reqData2.Ext, &reqDataExt2) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + var reqData3 *openrtb.BidRequest + error = json.Unmarshal(res[2].Body, &reqData3) + assert.NoError(t, error, "Response body unmarshalling error should be nil") + + var reqDataExt3 *appnexusReqExt + error = json.Unmarshal(reqData3.Ext, &reqDataExt3) + assert.NoError(t, error, "Response ext unmarshalling error should be nil") + + adPodId1 := reqDataExt1.Appnexus.AdPodId + adPodId2 := reqDataExt2.Appnexus.AdPodId + adPodId3 := reqDataExt3.Appnexus.AdPodId + + podIds := make(map[string]int) + podIds[adPodId1] = podIds[adPodId1] + 1 + podIds[adPodId2] = podIds[adPodId2] + 1 + podIds[adPodId3] = podIds[adPodId3] + 1 + + assert.Len(t, podIds, 2, "Incorrect number of unique pod ids") +} + // ---------------------------------------------------------------------------- // Code below this line tests the legacy, non-openrtb code flow. It can be deleted after we // clean up the existing code and make everything openrtb. diff --git a/adapters/appnexus/appnexusplatformtest/video/simple-video.json b/adapters/appnexus/appnexusplatformtest/video/simple-video.json deleted file mode 100644 index 7ee192be2c1..00000000000 --- a/adapters/appnexus/appnexusplatformtest/video/simple-video.json +++ /dev/null @@ -1,132 +0,0 @@ -{ - "mockBidRequest": { - "id": "test-request-id", - "imp": [ - { - "id": "test-imp-id", - "video": { - "mimes": ["video/mp4"], - "minduration": 15, - "maxduration": 30, - "protocols": [2, 3, 5, 6, 7, 8], - "w": 940, - "h": 560 - }, - "ext": { - "bidder": { - "placement_id": 1 - } - } - } - ] - }, - - "httpCalls": [ - { - "expectedRequest": { - "uri": "http://ib.adnxs.com/openrtb2", - "body": { - "id": "test-request-id", - "ext": { - "appnexus": { - "hb_source": 9 - }, - "prebid": {} - }, - "imp": [ - { - "id": "test-imp-id", - "video": { - "mimes": ["video/mp4"], - "minduration": 15, - "maxduration": 30, - "protocols": [2, 3, 5, 6, 7, 8], - "w": 940, - "h": 560 - }, - "ext": { - "appnexus": { - "placement_id": 1 - } - } - } - ] - } - }, - "mockResponse": { - "status": 200, - "body": { - "id": "test-request-id", - "seatbid": [ - { - "seat": "958", - "bid": [{ - "id": "7706636740145184841", - "impid": "test-imp-id", - "price": 0.500000, - "adid": "29681110", - "adm": "some-test-ad", - "adomain": ["appnexus.com"], - "iurl": "http://nym1-ib.adnxs.com/cr?id=29681110", - "cid": "958", - "crid": "29681110", - "h": 250, - "w": 300, - "cat": ["IAB9-1"], - "ext": { - "appnexus": { - "brand_id": 9, - "brand_category_id": 9, - "auction_id": 8189378542222915032, - "bid_ad_type": 1, - "bidder_id": 2, - "ranking_price": 0.000000, - "deal_priority": 5 - } - } - }] - } - ], - "bidid": "5778926625248726496", - "cur": "USD" - } - } - } - ], - - "expectedBidResponses": [ - { - "currency": "USD", - "bids": [ - { - "bid": { - "id": "7706636740145184841", - "impid": "test-imp-id", - "price": 0.5, - "adm": "some-test-ad", - "adid": "29681110", - "adomain": ["appnexus.com"], - "iurl": "http://nym1-ib.adnxs.com/cr?id=29681110", - "cid": "958", - "crid": "29681110", - "w": 300, - "h": 250, - "cat": ["IAB5-3"], - "ext": { - "appnexus": { - "brand_id": 9, - "brand_category_id": 9, - "auction_id": 8189378542222915032, - "bid_ad_type": 1, - "bidder_id": 2, - "ranking_price": 0.000000, - "deal_priority": 5 - } - } - }, - "type": "video" - } - ] - } - ] - } \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/exemplary/banner-app.json b/adapters/audienceNetwork/audienceNetworktest/exemplary/banner-app.json new file mode 100644 index 00000000000..3ac62d90cd4 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/exemplary/banner-app.json @@ -0,0 +1,116 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "httpcalls": [{ + "expectedRequest": { + "uri": "https://an.facebook.com/placementbid.ortb", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Fb-Pool-Routing-Token": [ + "v4_bidder_token" + ] + }, + "body": { + "id": "test-imp-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "w": -1, + "h": 250 + }, + "tagid": "123_456" + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid", + "publisher": { + "id": "123" + } + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500, + "ext": { + "authentication_id": "4e24a2b23fbfb5e41a9093b921d6cddf497c24dd5f63879038cec2ab2f27d174", + "platformid": "test-platform-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-imp-id", + "seatbid": [{ + "bid": [{ + "id": "987", + "impid": "test-imp-id", + "price": 1.000000, + "adm": "{\"type\":\"ID\",\"bid_id\":\"987\",\"placement_id\":\"123_456\",\"resolved_placement_id\":\"123_456\",\"sdk_version\":\"5.5.0\",\"device_id\":\"abc\",\"template\":1,\"payload\":null,\"bid_time_token\":\"v4_bidder_token=\"}", + "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" + }] + }], + "bidid": "654", + "cur": "USD" + } + } + }], + "expectedBidResponses": [{ + "currency": "USD", + "bids": [{ + "bid": { + "id": "987", + "impid": "test-imp-id", + "price": 1, + "adm": "{\"type\":\"ID\",\"bid_id\":\"987\",\"placement_id\":\"123_456\",\"resolved_placement_id\":\"123_456\",\"sdk_version\":\"5.5.0\",\"device_id\":\"abc\",\"template\":1,\"payload\":null,\"bid_time_token\":\"v4_bidder_token=\"}", + "adid": "987", + "crid": "987", + "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" + }, + "type": "banner" + }] + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/exemplary/banner.json b/adapters/audienceNetwork/audienceNetworktest/exemplary/banner.json deleted file mode 100644 index f5f92515e26..00000000000 --- a/adapters/audienceNetwork/audienceNetworktest/exemplary/banner.json +++ /dev/null @@ -1,138 +0,0 @@ -{ - "mockBidRequest": { - "id": "test-req-id", - "imp": [ - { - "id": "test-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 250 - } - ], - "w": 300, - "h": 250 - }, - "ext": { - "bidder": { - "publisherid": "123", - "placementid": "456" - } - } - } - ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" - }, - "device": { - "ip": "152.193.6.74" - }, - "user": { - "id": "db089de9-a62e-4861-a881-0ff15e052516", - "buyeruid": "v4_bidder_token" - }, - "tmax": 500 - }, - "httpcalls": [ - { - "expectedRequest": { - "uri": "https://an.facebook.com/placementbid.ortb", - "headers": { - "Accept": [ - "application/json" - ], - "Content-Type": [ - "application/json;charset=utf-8" - ], - "X-Fb-Pool-Routing-Token": [ - "v4_bidder_token" - ] - }, - "body": { - "id": "test-imp-id", - "imp": [ - { - "id": "test-imp-id", - "banner": { - "w": -1, - "h": 250 - }, - "tagid": "123_456" - } - ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", - "publisher": { - "id": "123" - } - }, - "device": { - "ip": "152.193.6.74" - }, - "user": { - "id": "db089de9-a62e-4861-a881-0ff15e052516", - "buyeruid": "v4_bidder_token" - }, - "tmax": 500, - "ext": { - "authentication_id": "4e24a2b23fbfb5e41a9093b921d6cddf497c24dd5f63879038cec2ab2f27d174", - "platformid": "test-platform-id" - } - } - }, - "mockResponse": { - "status": 200, - "body": { - "id": "test-imp-id", - "seatbid": [ - { - "bid": [ - { - "id": "987", - "impid": "test-imp-id", - "price": 1.000000, - "adm": "{\"type\":\"ID\",\"bid_id\":\"987\",\"placement_id\":\"123_456\",\"resolved_placement_id\":\"123_456\",\"sdk_version\":\"5.5.0\",\"device_id\":\"abc\",\"template\":1,\"payload\":null,\"bid_time_token\":\"v4_bidder_token=\"}", - "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", - "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", - "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" - } - ] - } - ], - "bidid": "654", - "cur": "USD" - } - } - } - ], - "expectedBidResponses": [ - { - "currency": "USD", - "bids": [ - { - "bid": { - "id": "987", - "impid": "test-imp-id", - "price": 1, - "adm": "{\"type\":\"ID\",\"bid_id\":\"987\",\"placement_id\":\"123_456\",\"resolved_placement_id\":\"123_456\",\"sdk_version\":\"5.5.0\",\"device_id\":\"abc\",\"template\":1,\"payload\":null,\"bid_time_token\":\"v4_bidder_token=\"}", - "adid": "987", - "crid": "987", - "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", - "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", - "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" - }, - "type": "banner" - } - ] - } - ] -} diff --git a/adapters/audienceNetwork/audienceNetworktest/exemplary/interstitial.json b/adapters/audienceNetwork/audienceNetworktest/exemplary/interstitial.json index bad228d5f18..573032c81e1 100644 --- a/adapters/audienceNetwork/audienceNetworktest/exemplary/interstitial.json +++ b/adapters/audienceNetwork/audienceNetworktest/exemplary/interstitial.json @@ -23,9 +23,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -64,15 +64,9 @@ "tagid": "123_456" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "123" } diff --git a/adapters/audienceNetwork/audienceNetworktest/exemplary/native-1.1.json b/adapters/audienceNetwork/audienceNetworktest/exemplary/native-1.1.json index 9090d80d099..08639bee013 100644 --- a/adapters/audienceNetwork/audienceNetworktest/exemplary/native-1.1.json +++ b/adapters/audienceNetwork/audienceNetworktest/exemplary/native-1.1.json @@ -16,9 +16,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -56,15 +56,9 @@ "tagid": "123_456" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "123" } diff --git a/adapters/audienceNetwork/audienceNetworktest/exemplary/video.json b/adapters/audienceNetwork/audienceNetworktest/exemplary/video.json index 22c62f8b821..35bdf9a443e 100644 --- a/adapters/audienceNetwork/audienceNetworktest/exemplary/video.json +++ b/adapters/audienceNetwork/audienceNetworktest/exemplary/video.json @@ -21,9 +21,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -66,15 +66,9 @@ "tagid": "123_456" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "123" } diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/banner-format-only.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/banner-format-only.json index 3edd6569258..450e0d9e45b 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/banner-format-only.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/banner-format-only.json @@ -24,9 +24,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -64,15 +64,9 @@ "tagid": "123_456" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "123" } diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-adm.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-adm.json new file mode 100644 index 00000000000..c33807bda74 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-adm.json @@ -0,0 +1,103 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "httpcalls": [{ + "expectedRequest": { + "uri": "https://an.facebook.com/placementbid.ortb", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Fb-Pool-Routing-Token": [ + "v4_bidder_token" + ] + }, + "body": { + "id": "test-imp-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "w": -1, + "h": 250 + }, + "tagid": "123_456" + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid", + "publisher": { + "id": "123" + } + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500, + "ext": { + "authentication_id": "4e24a2b23fbfb5e41a9093b921d6cddf497c24dd5f63879038cec2ab2f27d174", + "platformid": "test-platform-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-imp-id", + "seatbid": [{ + "bid": [{ + "id": "987", + "impid": "test-imp-id", + "price": 1.000000, + "adm": "malformed", + "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" + }] + }], + "bidid": "654", + "cur": "USD" + } + } + }], + "expectedMakeBidsErrors": [{ + "value": "invalid character 'm' looking for beginning of value", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-banner-height.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-banner-height.json index fa9fd9132b8..b229d41a27a 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-banner-height.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-banner-height.json @@ -22,9 +22,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-interstitial.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-interstitial.json new file mode 100644 index 00000000000..68ca8044812 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/invalid-interstitial.json @@ -0,0 +1,40 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "video": { + "mimes": ["video/mp4"], + "minduration": 15, + "maxduration": 30, + "protocols": [2, 3, 5, 6, 7, 8], + "linearity": 1, + "w": 940, + "h": 560 + }, + "instl": 1, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "expectedMakeRequestsErrors": [{ + "value": "imp #test-imp-id: interstitial imps are only supported for banner", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm-bidid.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm-bidid.json new file mode 100644 index 00000000000..50212155752 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm-bidid.json @@ -0,0 +1,107 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "httpcalls": [{ + "expectedRequest": { + "uri": "https://an.facebook.com/placementbid.ortb", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Fb-Pool-Routing-Token": [ + "v4_bidder_token" + ] + }, + "body": { + "id": "test-imp-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "w": -1, + "h": 250 + }, + "tagid": "123_456" + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid", + "publisher": { + "id": "123" + } + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500, + "ext": { + "authentication_id": "4e24a2b23fbfb5e41a9093b921d6cddf497c24dd5f63879038cec2ab2f27d174", + "platformid": "test-platform-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-imp-id", + "seatbid": [{ + "bid": [{ + "id": "987", + "impid": "test-imp-id", + "price": 1.000000, + "adm": "{\"type\":\"ID\",\"placement_id\":\"123_456\",\"resolved_placement_id\":\"123_456\",\"sdk_version\":\"5.5.0\",\"device_id\":\"abc\",\"template\":1,\"payload\":null,\"bid_time_token\":\"v4_bidder_token=\"}", + "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" + }] + }], + "bidid": "654", + "cur": "USD" + } + } + }], + "expectedBidResponses": [{ + "currency": "USD", + "bids": [] + }], + "expectedMakeBidsErrors": [{ + "value": "bid 987 missing 'bid_id' in 'adm'", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm.json new file mode 100644 index 00000000000..832b16dca22 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-adm.json @@ -0,0 +1,106 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "httpcalls": [{ + "expectedRequest": { + "uri": "https://an.facebook.com/placementbid.ortb", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Fb-Pool-Routing-Token": [ + "v4_bidder_token" + ] + }, + "body": { + "id": "test-imp-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "w": -1, + "h": 250 + }, + "tagid": "123_456" + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid", + "publisher": { + "id": "123" + } + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500, + "ext": { + "authentication_id": "4e24a2b23fbfb5e41a9093b921d6cddf497c24dd5f63879038cec2ab2f27d174", + "platformid": "test-platform-id" + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "test-imp-id", + "seatbid": [{ + "bid": [{ + "id": "987", + "impid": "test-imp-id", + "price": 1.000000, + "nurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=0&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "lurl": "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&ortb_loss_code=${AUCTION_LOSS}&clearing_price=${AUCTION_PRICE}&app_version=iOS-1.0", + "burl": "https://www.facebook.com/audiencenetwork/burl/?partner=test-platform-id&app=def&placement=456&auction=123&impression=123&request=123478&bid=987&clearing_price=${AUCTION_PRICE}" + }] + }], + "bidid": "654", + "cur": "USD" + } + } + }], + "expectedBidResponses": [{ + "currency": "USD", + "bids": [] + }], + "expectedMakeBidsErrors": [{ + "value": "Bid 987 missing 'adm'", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-banner-height.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-banner-height.json index 016e8de0ef0..0793f990049 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-banner-height.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/missing-banner-height.json @@ -20,9 +20,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/multi-imp.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/multi-imp.json index 16e8aede10c..682c33e46b8 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/multi-imp.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/multi-imp.json @@ -41,9 +41,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -81,15 +81,9 @@ "tagid": "pub1_plmt1" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "pub1" } @@ -158,15 +152,9 @@ "tagid": "pub2_plmt2" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "pub2" } diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/no-bid-204.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/no-bid-204.json index bb192aad76f..642e495810a 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/no-bid-204.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/no-bid-204.json @@ -16,9 +16,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -56,15 +56,9 @@ "tagid": "123_456" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "123" } diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/no-imps.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/no-imps.json new file mode 100644 index 00000000000..fccdf71ca4a --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/no-imps.json @@ -0,0 +1,22 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "expectedMakeRequestsErrors": [{ + "value": "No impressions provided", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/required-buyeruid.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/required-buyeruid.json index 964dcb48b48..72b4fbacdd1 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/required-buyeruid.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/required-buyeruid.json @@ -26,9 +26,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-placementId.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-placementId.json index a9c3c23d298..f13b70e1be2 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-placementId.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-placementId.json @@ -25,9 +25,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-publisherId.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-publisherId.json index c50f3d36378..a80a1e09b65 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-publisherId.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/required-param-publisherId.json @@ -25,9 +25,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/server-error-500.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/server-error-500.json new file mode 100644 index 00000000000..f0a11905cf8 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/server-error-500.json @@ -0,0 +1,87 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "native": { + "request": "{\"ver\":\"1.1\",\"context\":1,\"contextsubtype\":11,\"plcmttype\":4,\"plcmtcnt\":1,\"assets\":[{\"id\":1,\"required\":1,\"title\":{\"len\":500}},{\"id\":2,\"required\":1,\"img\":{\"type\":3,\"wmin\":1,\"hmin\":1}},{\"id\":3,\"required\":0,\"data\":{\"type\":1,\"len\":200}},{\"id\":4,\"required\":0,\"data\":{\"type\":2,\"len\":15000}},{\"id\":5,\"required\":0,\"data\":{\"type\":6,\"len\":40}},{\"id\":6,\"required\":0,\"data\":{\"type\":500}}]}", + "ver": "1.1" + }, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "httpcalls": [{ + "expectedRequest": { + "uri": "https://an.facebook.com/placementbid.ortb", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Fb-Pool-Routing-Token": [ + "v4_bidder_token" + ] + }, + "body": { + "id": "test-imp-id", + "imp": [{ + "id": "test-imp-id", + "native": { + "w": -1, + "h": -1 + }, + "tagid": "123_456" + }], + "app": { + "id": "app-abc", + "bundle": "com.prebid", + "publisher": { + "id": "123" + } + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500, + "ext": { + "authentication_id": "4e24a2b23fbfb5e41a9093b921d6cddf497c24dd5f63879038cec2ab2f27d174", + "platformid": "test-platform-id" + } + } + }, + "mockResponse": { + "headers": { + "X-Fb-An-Errors": [ + "someError" + ]}, + "status": 500 + } + }], + "expectedMakeBidsErrors": [{ + "value": "Unexpected status code 500 with error message 'someError'", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/site-not-supported.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/site-not-supported.json new file mode 100644 index 00000000000..9155352a192 --- /dev/null +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/site-not-supported.json @@ -0,0 +1,38 @@ +{ + "mockBidRequest": { + "id": "test-req-id", + "imp": [{ + "id": "test-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "publisherid": "123", + "placementid": "456" + } + } + }], + "site": { + "domain": "prebid.org", + "page": "prebid.org" + }, + "device": { + "ip": "152.193.6.74" + }, + "user": { + "id": "db089de9-a62e-4861-a881-0ff15e052516", + "buyeruid": "v4_bidder_token" + }, + "tmax": 500 + }, + "expectedMakeRequestsErrors": [{ + "value": "Site impressions are not supported.", + "comparison": "literal" + }] +} \ No newline at end of file diff --git a/adapters/audienceNetwork/audienceNetworktest/supplemental/split-placementId.json b/adapters/audienceNetwork/audienceNetworktest/supplemental/split-placementId.json index 4c561c55276..45c34192ea2 100644 --- a/adapters/audienceNetwork/audienceNetworktest/supplemental/split-placementId.json +++ b/adapters/audienceNetwork/audienceNetworktest/supplemental/split-placementId.json @@ -21,9 +21,9 @@ } } ], - "site": { - "domain": "prebid.org", - "page": "prebid.org" + "app": { + "id": "app-abc", + "bundle": "com.prebid" }, "device": { "ip": "152.193.6.74" @@ -50,15 +50,9 @@ "tagid": "123_456" } ], - "ext": { - "appnexus": { - "hb_source": 5 - }, - "prebid": {} - }, - "site": { - "domain": "prebid.org", - "page": "prebid.org", + "app": { + "id": "app-abc", + "bundle": "com.prebid", "publisher": { "id": "123" } diff --git a/adapters/audienceNetwork/facebook.go b/adapters/audienceNetwork/facebook.go index 3bc072a8385..0759a09d80b 100644 --- a/adapters/audienceNetwork/facebook.go +++ b/adapters/audienceNetwork/facebook.go @@ -10,16 +10,17 @@ import ( "net/http" "strings" - "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/adapters" "github.com/PubMatic-OpenWrap/prebid-server/errortypes" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/PubMatic-OpenWrap/prebid-server/util/maputil" + + "github.com/PubMatic-OpenWrap/openrtb" "github.com/buger/jsonparser" "github.com/golang/glog" ) type FacebookAdapter struct { - http *adapters.HTTPAdapter URI string nonSecureUri string platformID string @@ -35,15 +36,6 @@ var supportedBannerHeights = map[uint64]bool{ 250: true, } -// used for cookies and such -func (a *FacebookAdapter) Name() string { - return "audienceNetwork" -} - -func (a *FacebookAdapter) SkipNoCookies() bool { - return false -} - type facebookReqExt struct { PlatformID string `json:"platformid"` AuthID string `json:"authentication_id"` @@ -62,6 +54,12 @@ func (this *FacebookAdapter) MakeRequests(request *openrtb.BidRequest, reqInfo * }} } + if request.Site != nil { + return nil, []error{&errortypes.BadInput{ + Message: "Site impressions are not supported.", + }} + } + return this.buildRequests(request) } @@ -151,10 +149,6 @@ func (this *FacebookAdapter) modifyRequest(out *openrtb.BidRequest) error { app := *out.App app.Publisher = &openrtb.Publisher{ID: pubId} out.App = &app - } else { - site := *out.Site - site.Publisher = &openrtb.Publisher{ID: pubId} - out.Site = &site } if err = this.modifyImp(imp); err != nil { @@ -178,8 +172,10 @@ func (this *FacebookAdapter) modifyImp(out *openrtb.Imp) error { } } - switch impType { - case openrtb_ext.BidTypeBanner: + if impType == openrtb_ext.BidTypeBanner { + bannerCopy := *out.Banner + out.Banner = &bannerCopy + if out.Instl == 1 { out.Banner.W = openrtb.Uint64Ptr(0) out.Banner.H = openrtb.Uint64Ptr(0) @@ -212,7 +208,6 @@ func (this *FacebookAdapter) modifyImp(out *openrtb.Imp) error { /* This will get overwritten post-serialization */ out.Banner.W = openrtb.Uint64Ptr(0) out.Banner.Format = nil - break } return nil @@ -239,102 +234,106 @@ func (this *FacebookAdapter) extractPlacementAndPublisher(out *openrtb.Imp) (str } } - placementId := fbExt.PlacementId - publisherId := fbExt.PublisherId + placementID := fbExt.PlacementId + publisherID := fbExt.PublisherId // Support the legacy path with the caller was expected to pass in just placementId // which was an underscore concantenated string with the publisherId and placementId. // The new path for callers is to pass in the placementId and publisherId independently // and the below code will prefix the placementId that we pass to FAN with the publsiherId // so that we can abstract the implementation details from the caller - toks := strings.Split(placementId, "_") + toks := strings.Split(placementID, "_") if len(toks) == 1 { - if publisherId == "" { + if publisherID == "" { return "", "", &errortypes.BadInput{ Message: "Missing publisherId param", } } - return placementId, publisherId, nil + return placementID, publisherID, nil } else if len(toks) == 2 { - publisherId = toks[0] - placementId = toks[1] + publisherID = toks[0] + placementID = toks[1] } else { return "", "", &errortypes.BadInput{ - Message: fmt.Sprintf("Invalid placementId param '%s' and publisherId param '%s'", placementId, publisherId), + Message: fmt.Sprintf("Invalid placementId param '%s' and publisherId param '%s'", placementID, publisherID), } } - return placementId, publisherId, nil + return placementID, publisherID, nil } // XXX: This entire function is just a hack to get around mxmCherry 11.0.0 limitations, without // having to fork the library and maintain our own branch -func modifyImpCustom(json []byte, imp *openrtb.Imp) ([]byte, error) { +func modifyImpCustom(jsonData []byte, imp *openrtb.Imp) ([]byte, error) { impType, ok := resolveImpType(imp) if ok == false { panic("processing an invalid impression") } - var err error + var jsonMap map[string]interface{} + err := json.Unmarshal(jsonData, &jsonMap) + if err != nil { + return jsonData, err + } + + var impMap map[string]interface{} + if impSlice, ok := maputil.ReadEmbeddedSlice(jsonMap, "imp"); !ok { + return jsonData, errors.New("unable to find imp in json data") + } else if len(impSlice) == 0 { + return jsonData, errors.New("unable to find imp[0] in json data") + } else if impMap, ok = impSlice[0].(map[string]interface{}); !ok { + return jsonData, errors.New("unexpected type for imp[0] found in json data") + } switch impType { case openrtb_ext.BidTypeBanner: - // The current version of mxmCherry (11.0.0) repesents banner.w as unsigned - // integers, so setting a value of -1 is not possible which is why we have to do it + // The current version of mxmCherry (11.0.0) represents banner.w as an unsigned + // integer, so setting a value of -1 is not possible which is why we have to do it // post-serialization - - // The above does not apply to interstitial impressions - if imp.Instl == 1 { - break - } - - json, err = jsonparser.Set(json, []byte("-1"), "imp", "[0]", "banner", "w") - if err != nil { - return json, err + isInterstitial := imp.Instl == 1 + if !isInterstitial { + if bannerMap, ok := maputil.ReadEmbeddedMap(impMap, "banner"); ok { + bannerMap["w"] = json.RawMessage("-1") + } else { + return jsonData, errors.New("unable to find imp[0].banner in json data") + } } - break - case openrtb_ext.BidTypeVideo: // mxmCherry omits video.w/h if set to zero, so we need to force set those // fields to zero post-serialization for the time being - json, err = jsonparser.Set(json, []byte("0"), "imp", "[0]", "video", "w") - if err != nil { - return json, err + if videoMap, ok := maputil.ReadEmbeddedMap(impMap, "video"); ok { + videoMap["w"] = json.RawMessage("0") + videoMap["h"] = json.RawMessage("0") + } else { + return jsonData, errors.New("unable to find imp[0].video in json data") } - json, err = jsonparser.Set(json, []byte("0"), "imp", "[0]", "video", "h") - if err != nil { - return json, err + case openrtb_ext.BidTypeNative: + nativeMap, ok := maputil.ReadEmbeddedMap(impMap, "native") + if !ok { + return jsonData, errors.New("unable to find imp[0].video in json data") } - break - - case openrtb_ext.BidTypeNative: // Set w/h to -1 for native impressions based on the facebook native spec. // We have to set this post-serialization since the OpenRTB protocol doesn't - // actaully support w/h in the native object - json, err = jsonparser.Set(json, []byte("-1"), "imp", "[0]", "native", "w") - if err != nil { - return json, err - } - - json, err = jsonparser.Set(json, []byte("-1"), "imp", "[0]", "native", "h") - if err != nil { - return json, err - } + // actually support w/h in the native object + nativeMap["w"] = json.RawMessage("-1") + nativeMap["h"] = json.RawMessage("-1") // The FAN adserver does not expect the native request payload, all that information // is derived server side based on the placement ID. We need to remove these pieces of // information manually since OpenRTB (and thus mxmCherry) never omit native.request - json = jsonparser.Delete(json, "imp", "[0]", "native", "ver") - json = jsonparser.Delete(json, "imp", "[0]", "native", "request") - - break + delete(nativeMap, "ver") + delete(nativeMap, "request") } - return json, nil + if jsonReEncoded, err := json.Marshal(jsonMap); err == nil { + return jsonReEncoded, nil + } else { + return nil, fmt.Errorf("unable to encode json data (%v)", err) + } } func (this *FacebookAdapter) MakeBids(request *openrtb.BidRequest, adapterRequest *adapters.RequestData, response *adapters.ResponseData) (*adapters.BidderResponse, []error) { @@ -430,7 +429,7 @@ func resolveImpType(imp *openrtb.Imp) (openrtb_ext.BidType, bool) { return openrtb_ext.BidTypeBanner, false } -func NewFacebookBidder(client *http.Client, platformID string, appSecret string) adapters.Bidder { +func NewFacebookBidder(platformID string, appSecret string) adapters.Bidder { if platformID == "" { glog.Errorf("No facebook partnerID specified. Calls to the Audience Network will fail. Did you set adapters.facebook.platform_id in the app config?") return &adapters.MisconfiguredBidder{ @@ -447,11 +446,8 @@ func NewFacebookBidder(client *http.Client, platformID string, appSecret string) } } - a := &adapters.HTTPAdapter{Client: client} - return &FacebookAdapter{ - http: a, - URI: "https://an.facebook.com/placementbid.ortb", + URI: "https://an.facebook.com/placementbid.ortb", //for AB test nonSecureUri: "http://an.facebook.com/placementbid.ortb", platformID: platformID, @@ -474,15 +470,11 @@ func (fa *FacebookAdapter) MakeTimeoutNotification(req *adapters.RequestData) (* return &adapters.RequestData{}, []error{err} } - // The publisher ID is either in the app object or the site object, depending on the supply of the request so we need - // to check both + // The publisher ID is expected in the app object pubID, err = jsonparser.GetString(req.Body, "app", "publisher", "id") if err != nil { - pubID, err = jsonparser.GetString(req.Body, "site", "publisher", "id") - if err != nil { - return &adapters.RequestData{}, []error{ - errors.New("path [app|site].publisher.id not found in the request"), - } + return &adapters.RequestData{}, []error{ + errors.New("path app.publisher.id not found in the request"), } } diff --git a/adapters/audienceNetwork/facebook_test.go b/adapters/audienceNetwork/facebook_test.go index b4744dce211..8ff05118a35 100644 --- a/adapters/audienceNetwork/facebook_test.go +++ b/adapters/audienceNetwork/facebook_test.go @@ -1,6 +1,7 @@ package audienceNetwork import ( + "errors" "testing" "time" @@ -40,14 +41,14 @@ type FacebookExt struct { } func TestJsonSamples(t *testing.T) { - adapterstest.RunJSONBidderTest(t, "audienceNetworktest", NewFacebookBidder(nil, "test-platform-id", "test-app-secret")) + adapterstest.RunJSONBidderTest(t, "audienceNetworktest", NewFacebookBidder("test-platform-id", "test-app-secret")) } func TestMakeTimeoutNoticeApp(t *testing.T) { req := adapters.RequestData{ Body: []byte(`{"id":"1234","imp":[{"id":"1234"}],"app":{"publisher":{"id":"5678"}}}`), } - fba := NewFacebookBidder(nil, "test-platform-id", "test-app-secret") + fba := NewFacebookBidder("test-platform-id", "test-app-secret") tb, ok := fba.(adapters.TimeoutBidder) if !ok { @@ -60,11 +61,11 @@ func TestMakeTimeoutNoticeApp(t *testing.T) { assert.Equal(t, expectedUri, toReq.Uri, "Facebook timeout notification not returning the expected URI.") } -func TestMakeTimeoutNoticeSite(t *testing.T) { +func TestMakeTimeoutNoticeBadRequest(t *testing.T) { req := adapters.RequestData{ - Body: []byte(`{"id":"1234","imp":[{"id":"1234"}],"site":{"publisher":{"id":"5678"}}}`), + Body: []byte(`{"imp":[{{"id":"1234"}}`), } - fba := NewFacebookBidder(nil, "test-platform-id", "test-app-secret") + fba := NewFacebookBidder("test-platform-id", "test-app-secret") tb, ok := fba.(adapters.TimeoutBidder) if !ok { @@ -72,24 +73,29 @@ func TestMakeTimeoutNoticeSite(t *testing.T) { } toReq, err := tb.MakeTimeoutNotification(&req) - assert.Nil(t, err, "Facebook MakeTimeoutNotification() return an error %v", err) - expectedUri := "https://www.facebook.com/audiencenetwork/nurl/?partner=test-platform-id&app=5678&auction=1234&ortb_loss_code=2" - assert.Equal(t, expectedUri, toReq.Uri, "Facebook timeout notification not returning the expected URI.") + assert.Empty(t, toReq.Uri, "Facebook MakeTimeoutNotification() did not return nil", err) + assert.NotNil(t, err, "Facebook MakeTimeoutNotification() did not return an error") + } -func TestMakeTimeoutNoticeBadRequest(t *testing.T) { - req := adapters.RequestData{ - Body: []byte(`{"imp":[{{"id":"1234"}}`), - } - fba := NewFacebookBidder(nil, "test-platform-id", "test-app-secret") +func TestNewFacebookBidderMissingPlatformID(t *testing.T) { + result := NewFacebookBidder("", "anyAppSecret") - tb, ok := fba.(adapters.TimeoutBidder) - if !ok { - t.Error("Facebook adapter is not a TimeoutAdapter") + expected := &adapters.MisconfiguredBidder{ + Name: "audienceNetwork", + Error: errors.New("Audience Network is not configured properly on this Prebid Server deploy. If you believe this should work, contact the company hosting the service and tell them to check their configuration."), } - toReq, err := tb.MakeTimeoutNotification(&req) - assert.Empty(t, toReq.Uri, "Facebook MakeTimeoutNotification() did not return nil", err) - assert.NotNil(t, err, "Facebook MakeTimeoutNotification() did not return an error") + assert.Equal(t, expected, result) +} + +func TestNewFacebookBidderMissingAppSecret(t *testing.T) { + result := NewFacebookBidder("anyPlatformID", "") + + expected := &adapters.MisconfiguredBidder{ + Name: "audienceNetwork", + Error: errors.New("Audience Network is not configured properly on this Prebid Server deploy. If you believe this should work, contact the company hosting the service and tell them to check their configuration."), + } + assert.Equal(t, expected, result) } diff --git a/adapters/avocet/usersync_test.go b/adapters/avocet/usersync_test.go index be4890df91a..12b7901cc90 100644 --- a/adapters/avocet/usersync_test.go +++ b/adapters/avocet/usersync_test.go @@ -23,7 +23,7 @@ func TestAvocetSyncer(t *testing.T) { Consent: "ConsentString", }, CCPA: ccpa.Policy{ - Value: "PrivacyString", + Consent: "PrivacyString", }, }) diff --git a/adapters/beachfront/beachfronttest/exemplary/minimal-banner.json b/adapters/beachfront/beachfronttest/exemplary/minimal-banner.json index 51ce4e9295e..6672e2af91d 100644 --- a/adapters/beachfront/beachfronttest/exemplary/minimal-banner.json +++ b/adapters/beachfront/beachfronttest/exemplary/minimal-banner.json @@ -24,7 +24,6 @@ } ] }, - "httpCalls": [ { "expectedRequest": { @@ -57,38 +56,40 @@ "ua": "", "adapterName": "BF_PREBID_S2S", "adapterVersion": "0.9.0", - "user": { - } + "user": {} } }, "mockResponse": { "status": 200, "body": [ { - "crid":"crid_1", - "price":2.942808, - "w":300, - "h":250, - "slot":"div-gpt-ad-1460505748561-0", - "adm":"
", + "id": "some_test_ad_id_1", + "impid": "some_test_ad_id_1", + "ttl": 300, + "crid": "94395500", + "w": 300, + "price": 2.942808, + "adid": "94395500", + "h": 250 + }] + }, + { + "seat": "45678", + "bid": [{ + "adm": "00:00:15", + "id": "some_test_ad_id_2", + "impid": "some_test_ad_id_2", + "ttl": 300, + "crid": "9999999", + "w": 1020, + "price": 1, + "adid": "9999999", + "h": 1000 + } + ] + }], + "cur": "USD" + } + } + }], + + "expectedBids": [ + { + "bid": { + "adm": "
", + "id": "some_test_ad_id_1", + "impid": "some_test_ad_id_1", + "ttl": 300, + "crid": "94395500", + "w": 300, + "price": 2.942808, + "adid": "94395500", + "h": 250 + }, + "type": "banner" + }, + { + "bid": { + "adm": "00:00:15", + "id": "some_test_ad_id_2", + "impid": "some_test_ad_id_2", + "ttl": 300, + "crid": "9999999", + "w": 1020, + "price": 1, + "adid": "9999999", + "h": 1000 + }, + "type": "video" + } + ] +} + \ No newline at end of file diff --git a/adapters/emx_digital/emx_digitaltest/exemplary/banner-and-video-site.json b/adapters/emx_digital/emx_digitaltest/exemplary/banner-and-video-site.json new file mode 100644 index 00000000000..c2b20cf1c5d --- /dev/null +++ b/adapters/emx_digital/emx_digitaltest/exemplary/banner-and-video-site.json @@ -0,0 +1,200 @@ +{ + "mockBidRequest": { + "id": "some_test_auction", + "imp": [{ + "id": "some_test_ad_id_1", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "tagid": "25251" + } + } + }, + { + "id": "some_test_ad_id_2", + "video":{ + "mimes": [ + "video/mp4", + "application/javascript" + ], + "protocols":[ + 2, + 3, + 5, + 6 + ], + "w":640, + "h":480 + }, + "ext": { + "bidder": { + "tagid": "25251" + } + } + } + ], + "device": { + "ua": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36", + "ip": "123.123.123.123", + "dnt": 1 + }, + "site": { + "domain": "www.publisher.com", + "page": "http://www.publisher.com/awesome/site?with=some¶meters=here" + } + }, + + "httpCalls": [{ + "expectedRequest": { + "uri": "https://hb.emxdgt.com?t=1000&ts=2060541160", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ], + "Referer": [ + "http://www.publisher.com/awesome/site?with=some¶meters=here" + ], + "Dnt": [ + "1" + ], + "User-Agent": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36" + ] + }, + "body": { + "id": "some_test_auction", + "imp": [{ + "id": "some_test_ad_id_1", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "tagid": "25251" + } + }, + "tagid": "25251", + "secure": 0 + }, + { + "id": "some_test_ad_id_2", + "video":{ + "mimes": [ + "video/mp4", + "application/javascript" + ], + "protocols":[ + 2, + 3, + 5, + 6 + ], + "w":640, + "h":480 + }, + "ext": { + "bidder": { + "tagid": "25251" + } + }, + "tagid": "25251", + "secure": 0 + }], + "site": { + "domain": "www.publisher.com", + "page": "http://www.publisher.com/awesome/site?with=some¶meters=here" + }, + "device": { + "ua": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36", + "ip": "123.123.123.123", + "dnt": 1 + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "some_test_auction", + "seatbid": [{ + "seat": "12356", + "bid": [{ + "adm": "
", + "id": "some_test_ad_id_1", + "impid": "some_test_ad_id_1", + "ttl": 300, + "crid": "94395500", + "w": 300, + "price": 2.942808, + "adid": "94395500", + "h": 250 + }] + }, + { + "seat": "45678", + "bid": [{ + "adm": "00:00:15", + "id": "some_test_ad_id_2", + "impid": "some_test_ad_id_2", + "ttl": 300, + "crid": "9999999", + "w": 1020, + "price": 1, + "adid": "9999999", + "h": 1000 + } + ] + }], + "cur": "USD" + } + } + }], + + "expectedBids": [{ + "bid": { + "adm": "
", + "id": "some_test_ad_id_1", + "impid": "some_test_ad_id_1", + "ttl": 300, + "crid": "94395500", + "w": 300, + "price": 2.942808, + "adid": "94395500", + "h": 250 + }, + "type": "banner" + }, + { + "bid": { + "adm": "00:00:15", + "id": "some_test_ad_id_2", + "impid": "some_test_ad_id_2", + "ttl": 300, + "crid": "9999999", + "w": 1020, + "price": 1, + "adid": "9999999", + "h": 1000 + }, + "type": "video" + } + ] +} + \ No newline at end of file diff --git a/adapters/emx_digital/emx_digitaltest/exemplary/banner-app.json b/adapters/emx_digital/emx_digitaltest/exemplary/banner-app.json new file mode 100644 index 00000000000..8de90f52192 --- /dev/null +++ b/adapters/emx_digital/emx_digitaltest/exemplary/banner-app.json @@ -0,0 +1,119 @@ +{ + "mockBidRequest": { + "id": "some_test_auction", + "imp": [{ + "id": "some_test_ad_id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "tagid": "25251" + } + } + }], + "device": { + "ua": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36", + "ip": "123.123.123.123", + "dnt": 1 + }, + "app": { + "domain": "www.publisher.com", + "storeurl": "http://www.publisher.com/awesome/site?with=some¶meters=here" + } + }, + + "httpCalls": [{ + "expectedRequest": { + "uri": "https://hb.emxdgt.com?t=1000&ts=2060541160", + "headers": { + "Accept": [ + "application/json" + ], + "Content-Type": [ + "application/json;charset=utf-8" + ], + "X-Forwarded-For": [ + "123.123.123.123" + ], + "Dnt": [ + "1" + ], + "User-Agent": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36" + ] + }, + "body": { + "id": "some_test_auction", + "imp": [{ + "id": "some_test_ad_id", + "banner": { + "format": [{ + "w": 300, + "h": 250 + }], + "w": 300, + "h": 250 + }, + "ext": { + "bidder": { + "tagid": "25251" + } + }, + "tagid": "25251", + "secure": 0 + }], + "app": { + "domain": "www.publisher.com", + "storeurl": "http://www.publisher.com/awesome/site?with=some¶meters=here" + }, + "device": { + "ua": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36", + "ip": "123.123.123.123", + "dnt": 1 + } + } + }, + "mockResponse": { + "status": 200, + "body": { + "id": "some_test_auction", + "seatbid": [{ + "seat": "12356", + "bid": [{ + "adm": "
", - "adid": "107987536", - "adomain": [ - "appnexus.com" - ], - "iurl": "https://nym1-ib.adnxs.com/cr?id=107987536", - "cid": "3532", - "crid": "107987536", - "w": 600, - "h": 500, - "ext": { - "prebid": { - "type": "banner", - "video": { - "duration": 0, - "primary_category": "" - } - }, - "bidder": { - "appnexus": { - "brand_id": 1, - "auction_id": 7311907164510136364, - "bidder_id": 2, - "bid_ad_type": 0 - } - } - } - }] - }], - "cur": "USD", - "ext": { - "responsetimemillis": { - "appnexus": 10 - }, - "tmaxrequest": 500 - } -} -``` - -### OpenRTB Extensions - -#### Conventions - -OpenRTB 2.5 permits exchanges to define their own extensions to any object from the spec. -These fall under the `ext` field of JSON objects. - -If `ext` is defined on an object, Prebid Server uses the following conventions: - -1. `ext` in "request objects" uses `ext.prebid` and/or `ext.{anyBidderCode}`. -2. `ext` on "response objects" uses `ext.prebid` and/or `ext.bidder`. -The only exception here is the top-level `BidResponse`, because it's bidder-independent. - -`ext.{anyBidderCode}` and `ext.bidder` extensions are defined by bidders. -`ext.prebid` extensions are defined by Prebid Server. - -Exceptions are made for extensions with "standard" recommendations: - -- `request.user.ext.digitrust` -- To support Digitrust -- `request.regs.ext.gdpr` and `request.user.ext.consent` -- To support GDPR -- `request.regs.us_privacy` -- To support CCPA -- `request.site.ext.amp` -- To identify AMP as the request source -- `request.app.ext.source` and `request.app.ext.version` -- To support identifying the displaymanager/SDK in mobile apps. If given, we expect these to be strings. - -#### Bid Adjustments - -Bidders [are encouraged](../../developers/add-new-bidder.md) to make Net bids. However, there's no way for Prebid to enforce this. -If you find that some bidders use Gross bids, publishers can adjust for it with `request.ext.prebid.bidadjustmentfactors`: - -``` -{ - "ext": { - "prebid": { - "bidadjustmentfactors": { - "appnexus": 0.8, - "rubicon": 0.7 - } - } - } -} -``` - -This may also be useful for publishers who want to account for different discrepancies with different bidders. - -#### Targeting - -Targeting refers to strings which are sent to the adserver to -[make header bidding possible](http://prebid.org/overview/intro.html#how-does-prebid-work). - -`request.ext.prebid.targeting` is an optional property which causes Prebid Server -to set these params on the response at `response.seatbid[i].bid[j].ext.prebid.targeting`. - -**Request format** (optional param `request.ext.prebid.targeting`) - -``` -{ - "ext": { - "prebid": { - "targeting": { - "pricegranularity": { - "precision": 2, - "ranges": [{ - "max": 20.00, - "increment": 0.10 // This is equivalent to the deprecated "pricegranularity": "medium" - }] - }, - "includewinners": false, // Optional param defaulting to true - "includebidderkeys": false // Optional param defaulting to true - } - } - } -} -``` -The list of price granularity ranges must be given in order of increasing `max` values. If `precision` is omitted, it will default to `2`. The minimum of a range will be 0 or the previous `max`. Any cmp above the largest `max` will go in the `max` pricebucket. - -For backwards compatibility the following strings will also be allowed as price granularity definitions. There is no guarantee that these will be honored in the future. "One of ['low', 'med', 'high', 'auto', 'dense']" See [price granularity definitions](http://prebid.org/prebid-mobile/adops-price-granularity.html) - -One of "includewinners" or "includebidderkeys" must be true (both default to true if unset). If both were false, then no targeting keys would be set, which is better configured by omitting targeting altogether. - -MediaType PriceGranularity (PBS-Java only) - when a single OpenRTB request contains multiple impressions with different mediatypes, or a single impression supports multiple formats, the different mediatypes may need different price granularities. If `mediatypepricegranularity` is present, `pricegranularity` would only be used for any mediatypes not specified. - -``` -{ - "ext": { - "prebid": { - "targeting": { - "mediatypepricegranularity": { - "banner": { - "ranges": [ - {"max": 20, "increment": 0.5} - ] - }, - "video": { - "ranges": [ - {"max": 10, "increment": 1}, - {"max": 20, "increment": 2}, - {"max": 50, "increment": 5} - ] - } - } - }, - "includewinners": true - } - } -} -``` - -**Response format** (returned in `bid.ext.prebid.targeting`) - -``` -{ - "seatbid": [{ - "bid": [{ - ... - "ext": { - "prebid": { - "targeting": { - "hb_bidder_{bidderName}": "The seatbid.seat which contains this bid", - "hb_size_{bidderName}": "A string like '300x250' using bid.w and bid.h for this bid", - "hb_pb_{bidderName}": "The bid.cpm, rounded down based on the price granularity." - } - } - } - }] - }] -} -``` - -The winning bid for each `request.imp[i]` will also contain `hb_bidder`, `hb_size`, and `hb_pb` -(with _no_ {bidderName} suffix). To prevent these keys, set `request.ext.prebid.targeting.includeWinners` to false. - -**NOTE**: Targeting keys are limited to 20 characters. If {bidderName} is too long, the returned key -will be truncated to only include the first 20 characters. - -#### Cookie syncs - -Each Bidder should receive their own ID in the `request.user.buyeruid` property. -Prebid Server has three ways to populate this field. In order of priority: - -1. If the request payload contains `request.user.buyeruid`, then that value will be sent to all Bidders. -In most cases, this is probably a bad idea. - -2. The request payload can store a `buyeruid` for each Bidder by defining `request.user.ext.prebid.buyeruids` like so: - -``` -{ - "user": { - "ext": { - "prebid": { - "buyeruids": { - "appnexus": "some-appnexus-id", - "rubicon": "some-rubicon-id" - } - } - } - } -} -``` - -Prebid Server's core logic will preprocess the request so that each Bidder sees their own value in the `request.user.buyeruid` field. - -3. Prebid Server will use its Cookie to map IDs for each Bidder. - -If you're using [Prebid.js](https://github.com/prebid/Prebid.js), this is happening automatically. - -If you're using another client, you can populate the Cookie of the Prebid Server host with User IDs -for each Bidder by using the `/cookie_sync` endpoint, and calling the URLs that it returns in the response. - -#### Native Request - -For each native request, the `assets` object's `id` field must not be defined. Prebid Server will set this automatically, using the index of the asset in the array as the ID. - - -#### Bidder Aliases - -Requests can define Bidder aliases if they want to refer to a Bidder by a separate name. -This can be used to request bids from the same Bidder with different params. For example: - -``` -{ - "imp": [{ - "id": "some-impression-id", - "video": { - "mimes": ["video/mp4"] - }, - "ext": { - "appnexus": { - "placementId": 123 - }, - "districtm": { - "placementId": 456 - } - } - }], - "ext": { - "prebid": { - "aliases": { - "districtm": "appnexus" - } - } - } -} -``` - -For all intents and purposes, the alias will be treated as another Bidder. This new Bidder will behave exactly -like the original, except that the Response will contain separate SeatBids, and any Targeting keys -will be formed using the alias' name. - -If an alias overlaps with a core Bidder's name, then the alias will take precedence. -This prevents breaking API changes as new Bidders are added to the project. - -For example, if the Request defines an alias like this: - -``` - "aliases": { - "appnexus": "rubicon" - } -``` - -then any `imp.ext.appnexus` params will actually go to the **rubicon** adapter. -It will become impossible to fetch bids from AppNexus within that Request. - -#### Bidder Response Times - -`response.ext.responsetimemillis.{bidderName}` tells how long each bidder took to respond. -These can help quantify the performance impact of "the slowest bidder." - -#### Bidder Errors - -`response.ext.errors.{bidderName}` contains messages which describe why a request may be "suboptimal". -For example, suppose a `banner` and a `video` impression are offered to a bidder -which only supports `banner`. - -In cases like these, the bidder can ignore the `video` impression and bid on the `banner` one. -However, the publisher can improve performance by only offering impressions which the bidder supports. - -For example, a request may return this in `response.ext` - -``` -{ - "ext": { - "errors": { - "appnexus": [{ - "code": 2, - "message": "A hybrid Banner/Audio Imp was offered, but Appnexus doesn't support Audio." - }], - "rubicon": [{ - "code": 1, - "message": "The request exceeded the timeout allocated" - }] - } - } -} -``` - -The codes currently defined are: - -``` -0 NoErrorCode -1 TimeoutCode -2 BadInputCode -3 BadServerResponseCode -999 UnknownErrorCode -``` - -#### Debugging - -`response.ext.debug.httpcalls.{bidder}` will be populated **only if** `request.test` **was set to 1**. - -This contains info about every request and response sent by the bidder to its server. -It is only returned on `test` bids for performance reasons, but may be useful during debugging. - -`response.ext.debug.resolvedrequest` will be populated **only if** `request.test` **was set to 1**. - -This contains the request after the resolution of stored requests and implicit information (e.g. site domain, device user agent). - -#### Stored Requests - -`request.imp[i].ext.prebid.storedrequest` incorporates a [Stored Request](../../developers/stored-requests.md) from the server. - -A typical `storedrequest` value looks like this: - -``` -{ - "imp": [{ - "ext": { - "prebid": { - "storedrequest": { - "id": "some-id" - } - } - } - }] -} -``` - -For more information, see the docs for [Stored Requests](../../developers/stored-requests.md). - -#### Cache bids - -Bids can be temporarily cached on the server by sending the following data as `request.ext.prebid.cache`: - -``` -{ - "ext": { - "prebid": { - "cache": { - "bids": {}, - "vastxml": {} - } - } - } -} -``` - -Both `bids` and `vastxml` are optional, but one of the two is required if you want to cache bids. This property will have no effect -unless `request.ext.prebid.targeting` is also set in the request. - -If `bids` is present, Prebid Server will make a _best effort_ to include these extra -`bid.ext.prebid.targeting` keys: - -- `hb_cache_id`: On the highest overall Bid in each Imp. -- `hb_cache_id_{bidderName}`: On the highest Bid from {bidderName} in each Imp. - -Clients _should not assume_ that these keys will exist, just because they were requested, though. -If they exist, the value will be a UUID which can be used to fetch Bid JSON from [Prebid Cache](https://github.com/prebid/prebid-cache). -They may not exist if the host company's cache is full, having connection problems, or other issues like that. - -If `vastxml` is present, PBS will try to add analogous keys `hb_uuid` and `hb_uuid_{bidderName}`. -In addition to the caveats above, these will exist _only if the relevant Bids are for Video_. -If they exist, the values can be used to fetch the bid's VAST XML from Prebid Cache directly. - -These options are mainly intended for certain limited Prebid Mobile setups, where bids cannot be cached client-side. - -#### GDPR - -Prebid Server supports the IAB's GDPR recommendations, which can be found [here](https://iabtechlab.com/wp-content/uploads/2018/02/OpenRTB_Advisory_GDPR_2018-02.pdf). - -This adds two optional properties: - -- `request.user.ext.consent`: Is the consent string required by the IAB standards. -- `request.regs.ext.gdpr`: Is 0 if the caller believes that the user is *not* under GDPR, 1 if the user *is* under GDPR, and undefined if we're not certain. - -These fields will be forwarded to each Bidder, so they can decide how to process them. - -#### Interstitial support -Additional support for interstitials is enabled through the addition of two fields to the request: -device.ext.prebid.interstitial.minwidthperc and device.ext.interstial.minheightperc -The values will be numbers that indicate the minimum allowed size for the ad, as a percentage of the base side. For example, a width of 600 and "minwidthperc": 60 would allow ads with widths from 360 to 600 pixels inclusive. - -Example: -``` -{ - "imp": [{ - ... - "banner": { - ... - } - "instl": 1, - ... - }] - "device": { - ... - "h": 640, - "w": 320, - "ext": { - "prebid": { - "interstitial": { - "minwidthperc": 60, - "minheightperc": 60 - } - } - } - } -} -``` - -PBS receiving a request for an interstitial imp and these parameters set, it will rewrite the format object within the interstitial imp. If the format array's first object is a size, PBS will take it as the max size for the interstitial. If that size is 1x1, it will look up the device's size and use that as the max size. If the format is not present, it will also use the device size as the max size. (1x1 support so that you don't have to omit the format object to use the device size) -PBS with interstitial support will come preconfigured with a list of common ad sizes. Preferentially organized by weighing the larger and more common sizes first. But no guarantees to the ordering will be made. PBS will generate a new format list for the interstitial imp by traversing this list and picking the first 10 sizes that fall within the imp's max size and minimum percentage size. There will be no attempt to favor aspect ratios closer to the original size's aspect ratio. The limit of 10 is enforced to ensure we don't overload bidders with an overlong list. All the interstitial parameters will still be passed to the bidders, so they may recognize them and use their own size matching algorithms if they prefer. - -#### Currency Support - -To set the desired 'ad server currency', use the standard OpenRTB `cur` attribute. Note that Prebid Server only looks at the first currency in the array. - -``` - "cur": ["USD"] -``` - -If you want or need to define currency conversion rates (e.g. for currencies that your Prebid Server doesn't support), -define ext.prebid.currency.rates. (Currently supported in PBS-Java only) - -``` -"ext": { - "prebid": { - "currency": { - "rates": { - "USD": { "UAH": 24.47, "ETB": 32.04 } - } - } - } -} -``` - -If it exists, a rate defined in ext.prebid.currency.rates has the highest priority. -If a currency rate doesn't exist in the request, the external file will be used. - -#### Supply Chain Support - - -Basic supply chains are passed to Prebid Server on `source.ext.schain` and passed through to bid adapters. Prebid Server does not currently offer the ability to add a node to the supply chain. - -Bidder-specific schains (PBS-Java only): - -``` -ext.prebid.schains: [ - { bidders: ["bidderA"], schain: { SCHAIN OBJECT 1}}, - { bidders: ["*"], schain: { SCHAIN OBJECT 2}} -] -``` -In this scenario, Prebid Server sends the first schain object to `bidderA` and the second schain object to everyone else. - -If there's already an source.ext.schain and a bidder is named in ext.prebid.schains (or covered by the wildcard condition), ext.prebid.schains takes precedent. - -#### Rewarded Video (PBS-Java only) - -Rewarded video is a way to incentivize users to watch ads by giving them 'points' for viewing an ad. A Prebid Server -client can declare a given adunit as eligible for rewards by declaring `imp.ext.prebid.is_rewarded_inventory:1`. - -#### Stored Responses (PBS-Java only) - -While testing SDK and video integrations, it's important, but often difficult, to get consistent responses back from bidders that cover a range of scenarios like different CPM values, deals, etc. Prebid Server supports a debugging workflow in two ways: - -- a stored-auction-response that covers multiple bidder responses -- multiple stored-bid-responses at the bidder adapter level - -**Single Stored Auction Response ID** - -When a storedauctionresponse ID is specified: - -- the rest of the ext.prebid block is irrelevant and ignored -- nothing is sent to any bidder adapter for that imp -- the response retrieved from the stored-response-id is assumed to be the entire contents of the seatbid object corresponding to that impression. - -This request: -``` -{ - "test":1, - "tmax":500, - "id": "test-auction-id", - "app": { ... }, - "ext": { - "prebid": { - "targeting": {}, - "cache": { "bids": {} } - } - }, - "imp": [ - { - "id": "a", - "ext": { "prebid": { "storedauctionresponse": { "id": "1111111111" } } } - }, - { - "id": "b", - "ext": { "prebid": { "storedauctionresponse": { "id": "22222222222" } } } - } - ] -} -``` - -Will result in this response, assuming that the ids exist in the appropriate DB table read by Prebid Server: -``` -{ - "id": "test-auction-id", - "seatbid": [ - { - // BidderA bids from storedauctionresponse=1111111111 - // BidderA bids from storedauctionresponse=22222222 - }, - { - // BidderB bids from storedauctionresponse=1111111111 - // BidderB bids from storedauctionresponse=22222222 - } - ] -} -``` - -**Multiple Stored Bid Response IDs** - -In contrast to what's outlined above, this approach lets some real auctions take place while some bidders have test responses that still exercise bidder code. For example, this request: - -``` -{ - "test":1, - "tmax":500, - "id": "test-auction-id", - "app": { ... }, - "ext": { - "prebid": { - "targeting": {}, - "cache": { "bids": {} } - } - }, - "imp": [ - { - "id": "a", - "ext": { - "prebid": { - "storedbidresponse": [ - { "bidder": "BidderA", "id": "333333" }, - { "bidder": "BidderB", "id": "444444" }, - ] - } - } - }, - { - "id": "b", - "ext": { - "prebid": { - "storedbidresponse": [ - { "bidder": "BidderA", "id": "5555555" }, - { "bidder": "BidderB", "id": "6666666" }, - ] - } - } - } - ] -} -``` -Could result in this response: - -``` -{ - "id": "test-auction-id", - "seatbid": [ - { - "bid": [ - // contents of storedbidresponse=3333333 as parsed by bidderA adapter - // contents of storedbidresponse=5555555 as parsed by bidderA adapter - ] - }, - { - // contents of storedbidresponse=4444444 as parsed by bidderB adapter - // contents of storedbidresponse=6666666 as parsed by bidderB adapter - } - ] -} -``` - -Setting up the storedresponse DB entries is the responsibility of each Prebid Server host company. - -See Prebid.org troubleshooting pages for how to utilize this feature within the context of the browser. - - -#### User IDs (PBS-Java only) - -Prebid Server adapters can support the [Prebid.js User ID modules](http://prebid.org/dev-docs/modules/userId.html) by reading the following extensions and passing them through to their server endpoints: - -``` -{ - "user": { - "ext": { - "eids": [{ - "source": "adserver.org", - "uids": [{ - "id": "111111111111", - "ext": { - "rtiPartner": "TDID" - } - }] - }, - { - "source": "pubcommon", - "id":"11111111" - } - ], - "digitrust": { - "id": "11111111111", - "keyv": 4 - } - } - } -} -``` - -#### First Party Data Support (PBS-Java only) - -This is the Prebid Server version of the Prebid.js First Party Data feature. It's a standard way for the page (or app) to supply first party data and control which bidders have access to it. - -It specifies where in the OpenRTB request non-standard attributes should be passed. For example: - -``` -{ - "ext": { - "prebid": { - "data": { "bidders": [ "rubicon", "appnexus" ] } // these are the bidders allowed to see protected data - } - }, - "site": { - "keywords": "", - "search": "", - "ext": { - data: { GLOBAL CONTEXT DATA } // only seen by bidders named in ext.prebid.data.bidders[] - } - }, - "user": { - "keywords": "", - "gender": "", - "yob": 1999, - "geo": {}, - "ext": { - data: { GLOBAL USER DATA } // only seen by bidders named in ext.prebid.data.bidders[] - } - }, - "imp": [ - "ext": { - "context": { - "keywords": "", - "search": "", - "data": { ADUNIT SPECFIC CONTEXT DATA } // can be seen by all bidders - } - } - ] -``` - -Prebid Server enforces the data permissioning - -So before passing the values to the bidder adapters, core will: - -1. check for ext.prebid.data.bidders -1. if it exists, store it locally, but remove it from the OpenRTB before being sent to the adapters -1. As the OpenRTB request is being sent to each adapter: - 1. if ext.prebid.data.bidders exists in the original request, and this bidder is on the list then copy site.ext.data, app.ext.data, and user.ext.data to their bidder request -- otherwise don't copy those blocks - 1. copy other objects as normal - -Each adapter must be coded to read the values from these locations and pass it to their endpoints appropriately. - -### OpenRTB Ambiguities - -This section describes the ways in which Prebid Server **implements** OpenRTB spec ambiguous parts. - -- `request.cur`: If `request.cur` is not specified in the bid request, Prebid Server will consider it as being `USD` whereas OpenRTB spec doesn't mention any default currency for bid request. -```request.cur: ['USD'] // Default value if not set``` - - -### OpenRTB Differences - -This section describes the ways in which Prebid Server **breaks** the OpenRTB spec. - -#### Allowed Bidders - -Prebid Server returns a 400 on requests which define `wseat` or `bseat`. -We may add support for these in the future, if there's compelling need. - -Instead, an impression is only offered to a bidder if `bidrequest.imp[i].ext.{bidderName}` exists. - -This supports publishers who want to sell different impressions to different bidders. - -#### Deprecated Properties - -This endpoint returns a 400 if the request contains deprecated properties (e.g. `imp.wmin`, `imp.hmax`). - -The error message in the response should describe how to "fix" the request to make it legal. -If the message is unclear, please [log an issue](https://github.com/PubMatic-OpenWrap/prebid-server/issues) -or [submit a pull request](https://github.com/PubMatic-OpenWrap/prebid-server/pulls) to improve it. - -#### Determining Bid Security (http/https) - -In the OpenRTB spec, `request.imp[i].secure` says: - -> Flag to indicate if the impression requires secure HTTPS URL creative assets and markup, -> where 0 = non-secure, 1 = secure. If omitted, the secure state is unknown, but non-secure -> HTTP support can be assumed. - -In Prebid Server, an `https` request which does not define `secure` will be forwarded to Bidders with a `1`. -Publishers who run `https` sites and want insecure ads can still set this to `0` explicitly. - -### See also - -- [The OpenRTB 2.5 spec](https://www.iab.com/wp-content/uploads/2016/03/OpenRTB-API-Specification-Version-2-5-FINAL.pdf) diff --git a/docs/endpoints/setuid.md b/docs/endpoints/setuid.md deleted file mode 100644 index c1746806371..00000000000 --- a/docs/endpoints/setuid.md +++ /dev/null @@ -1,26 +0,0 @@ -# Saving User Syncs - -This endpoint is used during cookie syncs. For technical details, see the -[Cookie Sync developer docs](../developers/cookie-syncs.md). - -## `GET /setuid` - -This endpoint saves a UserID for a Bidder in the Cookie. Saved IDs will be recognized for 7 days before being considered "stale" and being re-synced. - -### Query Params - -- `bidder`: The FamilyName of the [Usersyncer](../../usersync/usersync.go) which is being synced. -- `uid`: The ID which the Bidder uses to recognize this user. If undefined, the UID for `bidder` will be deleted. -- `gdpr`: This should be `1` if GDPR is in effect, `0` if not, and undefined if the caller isn't sure -- `gdpr_consent`: This is required if `gdpr` is one, and optional (but encouraged) otherwise. If present, it should be an [unpadded base64-URL](https://tools.ietf.org/html/rfc4648#page-7) encoded [Vendor Consent String](https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/Consent%20string%20and%20vendor%20list%20formats%20v1.1%20Final.md#vendor-consent-string-format-). - -If the `gdpr` and `gdpr_consent` params are included, this endpoint will _not_ write a cookie unless: - -1. The Vendor ID set by the Prebid Server host company has permission to save cookies for that user. -2. The Prebid Server host company did not configure it to run with GDPR support. - -If in doubt, contact the company hosting Prebid Server and ask if they're GDPR-ready. - -### Sample request - -`GET http://prebid.site.com/setuid?bidder=adnxs&uid=12345&gdpr=1&gdpr_consent=BONciguONcjGKADACHENAOLS1rAHDAFAAEAASABQAMwAeACEAFw` diff --git a/docs/endpoints/status.md b/docs/endpoints/status.md deleted file mode 100644 index 0c252397423..00000000000 --- a/docs/endpoints/status.md +++ /dev/null @@ -1,9 +0,0 @@ -## `GET /status` - -This endpoint will return a 2xx response whenever Prebid Server is ready to serve requests. -Its exact response can be [configured](../developers/configuration.md) with the `status_response` -config option. For example, in `pbs.yaml`: - -```yaml -status_response: "ok" -``` diff --git a/endpoints/auction.go b/endpoints/auction.go index dd45be8df03..ff9d8f5a0ee 100644 --- a/endpoints/auction.go +++ b/endpoints/auction.go @@ -21,7 +21,7 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" pbc "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" "github.com/PubMatic-OpenWrap/prebid-server/privacy" - gdprPolicy "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" + gdprPrivacy "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" "github.com/PubMatic-OpenWrap/prebid-server/usersync" "github.com/golang/glog" "github.com/julienschmidt/httprouter" @@ -190,7 +190,7 @@ func (a *auction) recoverSafely(inner func(*pbs.PBSBidder, pbsmetrics.AdapterLab } } -func (a *auction) shouldUsersync(ctx context.Context, bidder openrtb_ext.BidderName, gdprPrivacyPolicy gdprPolicy.Policy) bool { +func (a *auction) shouldUsersync(ctx context.Context, bidder openrtb_ext.BidderName, gdprPrivacyPolicy gdprPrivacy.Policy) bool { switch gdprPrivacyPolicy.Signal { case "0": return true @@ -310,10 +310,7 @@ func sortBidsAddKeywordsMobile(bids pbs.PBSBidSlice, pbs_req *pbs.PBSRequest, pr // after sorting we need to add the ad targeting keywords for i, bid := range bar { // We should eventually check for the error and do something. - roundedCpm, err := exchange.GetCpmStringValue(bid.Price, openrtb_ext.PriceGranularityFromString(priceGranularitySetting)) - if err != nil { - glog.Error(err.Error()) - } + roundedCpm := exchange.GetPriceBucket(bid.Price, openrtb_ext.PriceGranularityFromString(priceGranularitySetting)) hbSize := "" if bid.Width != 0 && bid.Height != 0 { @@ -511,7 +508,7 @@ func (a *auction) processUserSync(req *pbs.PBSRequest, bidder *pbs.PBSBidder, bl if uid == "" { bidder.NoCookie = true privacyPolicies := privacy.Policies{ - GDPR: gdprPolicy.Policy{ + GDPR: gdprPrivacy.Policy{ Signal: req.ParseGDPR(), Consent: req.ParseConsent(), }, diff --git a/endpoints/auction_test.go b/endpoints/auction_test.go index 9c3b9878efa..e24e9454e12 100644 --- a/endpoints/auction_test.go +++ b/endpoints/auction_test.go @@ -387,11 +387,12 @@ func TestShouldUsersync(t *testing.T) { }, metricsEngine: nil, } - privacyPolicy := gdprPolicy.Policy{ + gdprPrivacyPolicy := gdprPolicy.Policy{ Signal: gdprApplies, Consent: consent, } - allowSyncs := deps.shouldUsersync(context.Background(), openrtb_ext.BidderAdform, privacyPolicy) + + allowSyncs := deps.shouldUsersync(context.Background(), openrtb_ext.BidderAdform, gdprPrivacyPolicy) if allowSyncs != expectAllow { t.Errorf("Expected syncs: %t, allowed syncs: %t", expectAllow, allowSyncs) } @@ -408,6 +409,7 @@ type auctionMockPermissions struct { allowHostCookies bool allowPI bool allowGeo bool + allowID bool } func (m *auctionMockPermissions) HostCookiesAllowed(ctx context.Context, consent string) (bool, error) { @@ -418,8 +420,8 @@ func (m *auctionMockPermissions) BidderSyncAllowed(ctx context.Context, bidder o return m.allowBidderSync, nil } -func (m *auctionMockPermissions) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) { - return m.allowPI, m.allowGeo, nil +func (m *auctionMockPermissions) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { + return m.allowPI, m.allowGeo, m.allowID, nil } func (m *auctionMockPermissions) AMPException() bool { diff --git a/endpoints/cookie_sync.go b/endpoints/cookie_sync.go index b75c5d29b65..35ba3cb14a7 100644 --- a/endpoints/cookie_sync.go +++ b/endpoints/cookie_sync.go @@ -19,7 +19,7 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" "github.com/PubMatic-OpenWrap/prebid-server/privacy" "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" - gdprPolicy "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" + gdprPrivacy "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" "github.com/PubMatic-OpenWrap/prebid-server/usersync" "github.com/buger/jsonparser" "github.com/golang/glog" @@ -110,24 +110,30 @@ func (deps *cookieSyncDeps) Endpoint(w http.ResponseWriter, r *http.Request, _ h } } + parsedReq.filterExistingSyncs(deps.syncers, userSyncCookie, needSyncupForSameSite) + + adapterSyncs := make(map[openrtb_ext.BidderName]bool) + // assume all bidders will be privacy blocked + for _, b := range parsedReq.Bidders { + adapterSyncs[openrtb_ext.BidderName(b)] = true + } + privacyPolicy := privacy.Policies{ - GDPR: gdprPolicy.Policy{ + GDPR: gdprPrivacy.Policy{ Signal: gdprToString(parsedReq.GDPR), Consent: parsedReq.Consent, }, CCPA: ccpa.Policy{ - Value: parsedReq.USPrivacy, + Consent: parsedReq.USPrivacy, }, } - parsedReq.filterExistingSyncs(deps.syncers, userSyncCookie, needSyncupForSameSite) + parsedReq.filterForGDPR(deps.syncPermissions) - adapterSyncs := make(map[openrtb_ext.BidderName]bool) - // assume all bidders will be privacy blocked - for _, b := range parsedReq.Bidders { - adapterSyncs[openrtb_ext.BidderName(b)] = true + if deps.enforceCCPA { + parsedReq.filterForCCPA() } - parsedReq.filterForPrivacy(deps.syncPermissions, privacyPolicy, deps.enforceCCPA) + // surviving bidders are not privacy blocked for _, b := range parsedReq.Bidders { adapterSyncs[openrtb_ext.BidderName(b)] = false @@ -264,12 +270,7 @@ func (req *cookieSyncRequest) filterExistingSyncs(valid map[openrtb_ext.BidderNa } } -func (req *cookieSyncRequest) filterForPrivacy(permissions gdpr.Permissions, privacyPolicies privacy.Policies, enforceCCPA bool) { - if enforceCCPA && privacyPolicies.CCPA.ShouldEnforce() { - req.Bidders = nil - return - } - +func (req *cookieSyncRequest) filterForGDPR(permissions gdpr.Permissions) { if req.GDPR != nil && *req.GDPR == 0 { return } @@ -287,6 +288,25 @@ func (req *cookieSyncRequest) filterForPrivacy(permissions gdpr.Permissions, pri } } +func (req *cookieSyncRequest) filterForCCPA() { + validBidders := make(map[string]struct{}) + for _, v := range openrtb_ext.BidderMap { + validBidders[v.String()] = struct{}{} + } + + ccpaPolicy := &ccpa.Policy{Consent: req.USPrivacy} + ccpaParsedPolicy, err := ccpaPolicy.Parse(validBidders) + + if err == nil { + for i := 0; i < len(req.Bidders); i++ { + if ccpaParsedPolicy.ShouldEnforce(req.Bidders[i]) { + req.Bidders = append(req.Bidders[:i], req.Bidders[i+1:]...) + i-- + } + } + } +} + // filterToLimit will enforce a max limit on cookiesyncs supplied, picking a random subset of syncs to get to the limit if over. func (req *cookieSyncRequest) filterToLimit() { if req.Limit <= 0 { diff --git a/endpoints/cookie_sync_test.go b/endpoints/cookie_sync_test.go index eef441b854f..b25d369226c 100644 --- a/endpoints/cookie_sync_test.go +++ b/endpoints/cookie_sync_test.go @@ -377,8 +377,8 @@ func (g *gdprPerms) BidderSyncAllowed(ctx context.Context, bidder openrtb_ext.Bi return ok, nil } -func (g *gdprPerms) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) { - return true, true, nil +func (g *gdprPerms) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { + return true, true, true, nil } func (g *gdprPerms) AMPException() bool { diff --git a/endpoints/currency_rates.go b/endpoints/currency_rates.go index 1e09f88a582..20d5ba9fc6c 100644 --- a/endpoints/currency_rates.go +++ b/endpoints/currency_rates.go @@ -24,7 +24,7 @@ type rateConverter interface { } // newCurrencyRatesInfo creates a new CurrencyRatesInfo instance. -func newCurrencyRatesInfo(rateConverter rateConverter) currencyRatesInfo { +func newCurrencyRatesInfo(rateConverter rateConverter, fetchingInterval time.Duration) currencyRatesInfo { currencyRatesInfo := currencyRatesInfo{ Active: false, @@ -44,7 +44,6 @@ func newCurrencyRatesInfo(rateConverter rateConverter) currencyRatesInfo { source := infos.Source() currencyRatesInfo.Source = &source - fetchingInterval := infos.FetchingInterval() currencyRatesInfo.FetchingInterval = &fetchingInterval lastUpdated := infos.LastUpdated() @@ -57,8 +56,8 @@ func newCurrencyRatesInfo(rateConverter rateConverter) currencyRatesInfo { } // NewCurrencyRatesEndpoint returns current currency rates applied by the PBS server. -func NewCurrencyRatesEndpoint(rateConverter rateConverter) http.HandlerFunc { - currencyRateInfo := newCurrencyRatesInfo(rateConverter) +func NewCurrencyRatesEndpoint(rateConverter rateConverter, fetchingInterval time.Duration) http.HandlerFunc { + currencyRateInfo := newCurrencyRatesInfo(rateConverter, fetchingInterval) return func(w http.ResponseWriter, _ *http.Request) { jsonOutput, err := json.Marshal(currencyRateInfo) diff --git a/endpoints/currency_rates_test.go b/endpoints/currency_rates_test.go index 5e43cec05bf..be1c3bcdf56 100644 --- a/endpoints/currency_rates_test.go +++ b/endpoints/currency_rates_test.go @@ -14,20 +14,21 @@ import ( func TestCurrencyRatesEndpoint(t *testing.T) { // Setup: var testCases = []struct { - input rateConverter - expectedBody string - expectedCode int - description string + inputConverter rateConverter + inputFetchingInterval time.Duration + expectedBody string + expectedCode int + description string }{ { nil, + time.Duration(0), `{"active": false}`, http.StatusOK, "case 1 - rate converter is nil", }, { newRateConverterMock( - 5*time.Minute, "https://sync.test.com", time.Date(2019, 3, 2, 12, 54, 56, 651387237, time.UTC), newConversionMock(&map[string]map[string]float64{ @@ -36,6 +37,7 @@ func TestCurrencyRatesEndpoint(t *testing.T) { }, }), ), + 5 * time.Minute, `{ "active": true, "source": "https://sync.test.com", @@ -52,11 +54,11 @@ func TestCurrencyRatesEndpoint(t *testing.T) { }, { newRateConverterMock( - time.Duration(0), "", time.Time{}, nil, ), + time.Duration(0), `{ "active": true, "source": "", @@ -70,12 +72,14 @@ func TestCurrencyRatesEndpoint(t *testing.T) { newRateConverterMockWithInfo( newUnmarshableConverterInfoMock(), ), + time.Duration(0), "", http.StatusInternalServerError, "case 4 - invalid rates input for marshaling", }, { newRateConverterMockWithNilInfo(), + time.Duration(0), `{ "active": true }`, @@ -86,7 +90,7 @@ func TestCurrencyRatesEndpoint(t *testing.T) { for _, tc := range testCases { - handler := NewCurrencyRatesEndpoint(tc.input) + handler := NewCurrencyRatesEndpoint(tc.inputConverter, tc.inputFetchingInterval) w := httptest.NewRecorder() // Execute: @@ -117,21 +121,16 @@ func newConversionMock(rates *map[string]map[string]float64) *conversionMock { } type converterInfoMock struct { - source string - fetchingInterval time.Duration - lastUpdated time.Time - rates *map[string]map[string]float64 - additionalInfo interface{} + source string + lastUpdated time.Time + rates *map[string]map[string]float64 + additionalInfo interface{} } func (m converterInfoMock) Source() string { return m.source } -func (m converterInfoMock) FetchingInterval() time.Duration { - return m.fetchingInterval -} - func (m converterInfoMock) LastUpdated() time.Time { return m.lastUpdated } @@ -150,10 +149,6 @@ func (m unmarshableConverterInfoMock) Source() string { return "" } -func (m unmarshableConverterInfoMock) FetchingInterval() time.Duration { - return time.Duration(0) -} - func (m unmarshableConverterInfoMock) LastUpdated() time.Time { return time.Time{} } @@ -172,7 +167,6 @@ func newUnmarshableConverterInfoMock() unmarshableConverterInfoMock { } type rateConverterMock struct { - fetchingInterval time.Duration syncSourceURL string rates *conversionMock lastUpdated time.Time @@ -197,23 +191,20 @@ func (m rateConverterMock) GetInfo() currencies.ConverterInfo { rates = m.rates.GetRates() } return converterInfoMock{ - source: m.syncSourceURL, - fetchingInterval: m.fetchingInterval, - lastUpdated: m.lastUpdated, - rates: rates, + source: m.syncSourceURL, + lastUpdated: m.lastUpdated, + rates: rates, } } func newRateConverterMock( - fetchingInterval time.Duration, syncSourceURL string, lastUpdated time.Time, rates *conversionMock) rateConverterMock { return rateConverterMock{ - fetchingInterval: fetchingInterval, - syncSourceURL: syncSourceURL, - rates: rates, - lastUpdated: lastUpdated, + syncSourceURL: syncSourceURL, + rates: rates, + lastUpdated: lastUpdated, } } diff --git a/endpoints/events/account_test.go b/endpoints/events/account_test.go new file mode 100644 index 00000000000..559b39d096c --- /dev/null +++ b/endpoints/events/account_test.go @@ -0,0 +1,161 @@ +package events + +import ( + "errors" + "fmt" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/julienschmidt/httprouter" + "github.com/stretchr/testify/assert" + "io/ioutil" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestHandleAccountServiceErrors(t *testing.T) { + tests := map[string]struct { + fetcher *mockAccountsFetcher + cfg *config.Configuration + want struct { + code int + response string + } + }{ + "badRequest": { + fetcher: &mockAccountsFetcher{ + Fail: true, + Error: errors.New("some error"), + }, + cfg: &config.Configuration{ + AccountDefaults: config.Account{Disabled: true}, + AccountRequired: true, + MaxRequestSize: maxSize, + VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: false, + }, + }, + want: struct { + code int + response string + }{ + code: 400, + response: "Invalid request: some error\nInvalid request: Prebid-server could not verify the Account ID. Please reach out to the prebid server host.\n", + }, + }, + "serviceUnavailable": { + fetcher: &mockAccountsFetcher{ + Fail: false, + }, + cfg: &config.Configuration{ + BlacklistedAcctMap: map[string]bool{"testacc": true}, + MaxRequestSize: maxSize, + VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: false, + }, + }, + want: struct { + code int + response string + }{ + code: 503, + response: "Invalid request: Prebid-server has disabled Account ID: testacc, please reach out to the prebid server host.\n", + }, + }, + "timeout": { + fetcher: &mockAccountsFetcher{ + Fail: false, + DurationMS: 50, + }, + cfg: &config.Configuration{ + AccountDefaults: config.Account{Disabled: true}, + AccountRequired: true, + Event: config.Event{ + TimeoutMS: 1, + }, + MaxRequestSize: maxSize, + VTrack: config.VTrack{ + TimeoutMS: int64(1), + AllowUnknownBidder: false, + }, + }, + want: struct { + code int + response string + }{ + code: 504, + response: "Invalid request: context deadline exceeded\nInvalid request: Prebid-server could not verify the Account ID. Please reach out to the prebid server host.\n", + }, + }, + } + + for name, test := range tests { + + handlers := []struct { + name string + h httprouter.Handle + r *http.Request + }{ + vast(t, test.cfg, test.fetcher), + event(test.cfg, test.fetcher), + } + + for _, handler := range handlers { + t.Run(handler.name+"-"+name, func(t *testing.T) { + test.cfg.MarshalAccountDefaults() + + recorder := httptest.NewRecorder() + + // execute + handler.h(recorder, handler.r, nil) + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, test.want.code, recorder.Result().StatusCode, fmt.Sprintf("Expected %d", test.want.code)) + assert.Equal(t, test.want.response, string(d)) + }) + } + } +} + +func event(cfg *config.Configuration, fetcher stored_requests.AccountFetcher) struct { + name string + h httprouter.Handle + r *http.Request +} { + return struct { + name string + h httprouter.Handle + r *http.Request + }{ + name: "event", + h: NewEventEndpoint(cfg, fetcher, nil), + r: httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=b&x=1&a=testacc", strings.NewReader("")), + } +} + +func vast(t *testing.T, cfg *config.Configuration, fetcher stored_requests.AccountFetcher) struct { + name string + h httprouter.Handle + r *http.Request +} { + vtrackBody, err := getValidVTrackRequestBody(true, true) + if err != nil { + t.Fatal(err) + } + + return struct { + name string + h httprouter.Handle + r *http.Request + }{ + name: "vast", + h: NewVTrackEndpoint(cfg, fetcher, &vtrackMockCacheClient{}, adapters.BidderInfos{}), + r: httptest.NewRequest("POST", "/vtrack?a=testacc", strings.NewReader(vtrackBody)), + } +} diff --git a/endpoints/events/event.go b/endpoints/events/event.go new file mode 100644 index 00000000000..da18b16bd53 --- /dev/null +++ b/endpoints/events/event.go @@ -0,0 +1,338 @@ +package events + +import ( + "context" + "errors" + "fmt" + accountService "github.com/PubMatic-OpenWrap/prebid-server/account" + "github.com/PubMatic-OpenWrap/prebid-server/analytics" + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/julienschmidt/httprouter" + "net/http" + "net/url" + "strconv" + "time" +) + +const ( + // Required + TemplateUrl = "%v/event?t=%v&b=%v&a=%v" + TypeParameter = "t" + BidIdParameter = "b" + AccountIdParameter = "a" + + // Optional + BidderParameter = "bidder" + TimestampParameter = "ts" + FormatParameter = "f" + AnalyticsParameter = "x" +) + +var trackingPixelPng = &trackingPixel{ + Content: []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D, 0x49, 0x48, 0x44, + 0x52, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x08, 0x06, 0x00, 0x00, 0x00, 0x1F, 0x15, 0xC4, + 0x89, 0x00, 0x00, 0x00, 0x04, 0x73, 0x42, 0x49, 0x54, 0x08, 0x08, 0x08, 0x08, 0x7C, 0x08, 0x64, 0x88, + 0x00, 0x00, 0x00, 0x0D, 0x49, 0x44, 0x41, 0x54, 0x08, 0x99, 0x63, 0x60, 0x60, 0x60, 0x60, 0x00, 0x00, + 0x00, 0x05, 0x00, 0x01, 0x87, 0xA1, 0x4E, 0xD4, 0x00, 0x00, 0x00, 0x00, 0x49, 0x45, 0x4E, 0x44, 0xAE, + 0x42, 0x60, 0x82}, + ContentType: "image/png", +} + +type trackingPixel struct { + Content []byte `json:"content,omitempty"` + ContentType string `json:"content_type,omitempty"` +} + +type eventEndpoint struct { + Accounts stored_requests.AccountFetcher + Analytics analytics.PBSAnalyticsModule + Cfg *config.Configuration + TrackingPixel *trackingPixel +} + +func NewEventEndpoint(cfg *config.Configuration, accounts stored_requests.AccountFetcher, analytics analytics.PBSAnalyticsModule) httprouter.Handle { + ee := &eventEndpoint{ + Accounts: accounts, + Analytics: analytics, + Cfg: cfg, + TrackingPixel: trackingPixelPng, + } + + return ee.Handle +} + +func (e *eventEndpoint) Handle(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { + // parse event request from http req + eventRequest, errs := ParseEventRequest(r) + + // handle possible parsing errors + if len(errs) > 0 { + w.WriteHeader(http.StatusBadRequest) + + for _, err := range errs { + w.Write([]byte(fmt.Sprintf("invalid request: %s\n", err.Error()))) + } + + return + } + + // validate account id + accountId, err := checkRequiredParameter(r, AccountIdParameter) + + if err != nil { + w.WriteHeader(http.StatusUnauthorized) + w.Write([]byte(fmt.Sprintf("Account '%s' is required query parameter and can't be empty", AccountIdParameter))) + return + } + eventRequest.AccountID = accountId + + if eventRequest.Analytics != analytics.Enabled { + w.WriteHeader(http.StatusNoContent) + return + } + + ctx := context.Background() + if e.Cfg.Event.TimeoutMS > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, time.Duration(e.Cfg.Event.TimeoutMS)*time.Millisecond) + defer cancel() + } + + // get account details + account, errs := accountService.GetAccount(ctx, e.Cfg, e.Accounts, eventRequest.AccountID) + if len(errs) > 0 { + status, messages := HandleAccountServiceErrors(errs) + w.WriteHeader(status) + + for _, message := range messages { + w.Write([]byte(fmt.Sprintf("Invalid request: %s\n", message))) + } + return + } + + // account does not support events + if !account.EventsEnabled { + w.WriteHeader(http.StatusUnauthorized) + w.Write([]byte(fmt.Sprintf("Account '%s' doesn't support events", eventRequest.AccountID))) + return + } + + // handle notification event + e.Analytics.LogNotificationEventObject(&analytics.NotificationEvent{ + Request: eventRequest, + Account: account, + }) + + // Add tracking pixel if format == image + if eventRequest.Format == analytics.Image { + w.WriteHeader(http.StatusOK) + w.Header().Add("Content-Type", e.TrackingPixel.ContentType) + w.Write(e.TrackingPixel.Content) + + return + } + + w.WriteHeader(http.StatusNoContent) +} + +// EventRequestToUrl converts an analytics.EventRequest to an URL +func EventRequestToUrl(externalUrl string, request *analytics.EventRequest) string { + s := fmt.Sprintf(TemplateUrl, externalUrl, request.Type, request.BidID, request.AccountID) + + return s + optionalParameters(request) +} + +// ParseEventRequest parses an analytics.EventRequest from an Http request +func ParseEventRequest(r *http.Request) (*analytics.EventRequest, []error) { + event := &analytics.EventRequest{} + var errs []error + // validate type + if err := readType(event, r); err != nil { + errs = append(errs, err) + } + + // validate bidid + if bidid, err := checkRequiredParameter(r, BidIdParameter); err != nil { + errs = append(errs, err) + } else { + event.BidID = bidid + } + + // validate timestamp (optional) + if err := readTimestamp(event, r); err != nil { + errs = append(errs, err) + } + + // validate format (optional) + if err := readFormat(event, r); err != nil { + errs = append(errs, err) + } + + // validate analytics (optional) + if err := readAnalytics(event, r); err != nil { + errs = append(errs, err) + } + + // Bidder + event.Bidder = r.URL.Query().Get(BidderParameter) + + return event, errs +} + +// HandleAccountServiceErrors handles account.GetAccount errors +func HandleAccountServiceErrors(errs []error) (status int, messages []string) { + messages = []string{} + status = http.StatusBadRequest + + for _, er := range errs { + if errors.Is(er, context.DeadlineExceeded) { + er = &errortypes.Timeout{ + Message: er.Error(), + } + } + + messages = append(messages, er.Error()) + + errCode := errortypes.ReadCode(er) + + if errCode == errortypes.BlacklistedAppErrorCode || errCode == errortypes.BlacklistedAcctErrorCode { + status = http.StatusServiceUnavailable + } + + if errCode == errortypes.TimeoutErrorCode && status == http.StatusBadRequest { + status = http.StatusGatewayTimeout + } + } + + return status, messages +} + +func optionalParameters(request *analytics.EventRequest) string { + r := url.Values{} + + // timestamp + if request.Timestamp > 0 { + r.Add(TimestampParameter, strconv.FormatInt(request.Timestamp, 10)) + } + + // bidder + if request.Bidder != "" { + r.Add(BidderParameter, request.Bidder) + } + + // format + switch request.Format { + case analytics.Blank: + r.Add(FormatParameter, string(analytics.Blank)) + case analytics.Image: + r.Add(FormatParameter, string(analytics.Image)) + } + + //analytics + switch request.Analytics { + case analytics.Enabled: + r.Add(AnalyticsParameter, string(analytics.Enabled)) + case analytics.Disabled: + r.Add(AnalyticsParameter, string(analytics.Disabled)) + } + + opt := r.Encode() + + if opt != "" { + return "&" + opt + } + + return opt +} + +// readType validates analytics.EventRequest type +func readType(er *analytics.EventRequest, httpRequest *http.Request) error { + t, err := checkRequiredParameter(httpRequest, TypeParameter) + + if err != nil { + return err + } + + switch t { + case string(analytics.Imp): + er.Type = analytics.Imp + return nil + case string(analytics.Win): + er.Type = analytics.Win + return nil + default: + return &errortypes.BadInput{Message: fmt.Sprintf("unknown type: '%s'", t)} + } +} + +// readFormat validates analytics.EventRequest format attribute +func readFormat(er *analytics.EventRequest, httpRequest *http.Request) error { + f := httpRequest.URL.Query().Get(FormatParameter) + + if f != "" { + switch f { + case string(analytics.Blank): + er.Format = analytics.Blank + return nil + case string(analytics.Image): + er.Format = analytics.Image + return nil + default: + return &errortypes.BadInput{Message: fmt.Sprintf("unknown format: '%s'", f)} + } + } + + return nil +} + +// readAnalytics validates analytics.EventRequest analytics attribute +func readAnalytics(er *analytics.EventRequest, httpRequest *http.Request) error { + a := httpRequest.URL.Query().Get(AnalyticsParameter) + + if a != "" { + switch a { + case string(analytics.Enabled): + er.Analytics = analytics.Enabled + return nil + case string(analytics.Disabled): + er.Analytics = analytics.Disabled + return nil + default: + return &errortypes.BadInput{Message: fmt.Sprintf("unknown analytics: '%s'", a)} + } + } + + er.Analytics = analytics.Enabled + return nil +} + +// readTimestamp validates analytics.EventRequest timestamp attribute +func readTimestamp(er *analytics.EventRequest, httpRequest *http.Request) error { + t := httpRequest.URL.Query().Get(TimestampParameter) + + if t != "" { + ts, err := strconv.ParseInt(t, 10, 64) + + if err != nil { + return &errortypes.BadInput{Message: fmt.Sprintf("invalid request: error parsing timestamp '%s'", t)} + } + + er.Timestamp = ts + return nil + } + + return nil +} + +// checkRequiredParameter checks if http.Request contains all required parameters +func checkRequiredParameter(httpRequest *http.Request, parameter string) (string, error) { + t := httpRequest.URL.Query().Get(parameter) + + if t == "" { + return "", &errortypes.BadInput{Message: fmt.Sprintf("parameter '%s' is required", parameter)} + } + + return t, nil +} diff --git a/endpoints/events/event_test.go b/endpoints/events/event_test.go new file mode 100644 index 00000000000..d32d01ad562 --- /dev/null +++ b/endpoints/events/event_test.go @@ -0,0 +1,664 @@ +package events + +import ( + "context" + "encoding/base64" + "encoding/json" + "github.com/PubMatic-OpenWrap/prebid-server/analytics" + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/stretchr/testify/assert" + "io/ioutil" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +// Mock Analytics Module +type eventsMockAnalyticsModule struct { + Fail bool + Error error + Invoked bool +} + +func (e *eventsMockAnalyticsModule) LogAuctionObject(ao *analytics.AuctionObject) { + if e.Fail { + panic(e.Error) + } + return +} + +func (e *eventsMockAnalyticsModule) LogVideoObject(vo *analytics.VideoObject) { + if e.Fail { + panic(e.Error) + } + return +} + +func (e *eventsMockAnalyticsModule) LogCookieSyncObject(cso *analytics.CookieSyncObject) { + if e.Fail { + panic(e.Error) + } + return +} + +func (e *eventsMockAnalyticsModule) LogSetUIDObject(so *analytics.SetUIDObject) { + if e.Fail { + panic(e.Error) + } + return +} + +func (e *eventsMockAnalyticsModule) LogAmpObject(ao *analytics.AmpObject) { + if e.Fail { + panic(e.Error) + } + return +} + +func (e *eventsMockAnalyticsModule) LogNotificationEventObject(ne *analytics.NotificationEvent) { + if e.Fail { + panic(e.Error) + } + e.Invoked = true + + return +} + +// Mock Account fetcher +var mockAccountData = map[string]json.RawMessage{ + "events_enabled": json.RawMessage(`{"events_enabled":true}`), + "events_disabled": json.RawMessage(`{"events_enabled":false}`), +} + +type mockAccountsFetcher struct { + Fail bool + Error error + DurationMS int +} + +func (maf mockAccountsFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + if maf.DurationMS > 0 { + select { + case <-time.After(time.Duration(maf.DurationMS) * time.Millisecond): + break + case <-ctx.Done(): + return nil, []error{ctx.Err()} + } + } + + if account, ok := mockAccountData[accountID]; ok { + return account, nil + } + + if maf.Fail { + return nil, []error{maf.Error} + } + + return nil, []error{stored_requests.NotFoundError{accountID, "Account"}} +} + +// Tests + +func TestShouldReturnBadRequestWhenTypeIsMissing(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?b=test", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with missing type parameter") + assert.Equal(t, "invalid request: parameter 't' is required\n", string(d)) +} + +func TestShouldReturnBadRequestWhenTypeIsInvalid(t *testing.T) { + + // mock AccountsFetcher + mockAccounts := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=test&b=t", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccounts, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with invalid type parameter") + assert.Equal(t, "invalid request: unknown type: 'test'\n", string(d)) +} + +func TestShouldReturnBadRequestWhenBidIdIsMissing(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with missing bidid parameter") + assert.Equal(t, "invalid request: parameter 'b' is required\n", string(d)) +} + +func TestShouldReturnBadRequestWhenTimestampIsInvalid(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=q", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with invalid timestamp parameter") + assert.Equal(t, "invalid request: invalid request: error parsing timestamp 'q'\n", string(d)) +} + +func TestShouldReturnUnauthorizedWhenAccountIsMissing(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 401, recorder.Result().StatusCode, "Expected 401 on request with missing account id parameter") + assert.Equal(t, "Account 'a' is required query parameter and can't be empty", string(d)) +} + +func TestShouldReturnBadRequestWhenFormatValueIsInvalid(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=q", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with invalid format parameter") + assert.Equal(t, "invalid request: unknown format: 'q'\n", string(d)) +} + +func TestShouldReturnBadRequestWhenAnalyticsValueIsInvalid(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=b&x=4", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with invalid analytics parameter") + assert.Equal(t, "invalid request: unknown analytics: '4'\n", string(d)) +} + +func TestShouldNotPassEventToAnalyticsReporterWhenAccountNotFound(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: true, + Error: stored_requests.NotFoundError{ID: "testacc"}, + } + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=b&x=1&a=testacc", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 401, recorder.Result().StatusCode, "Expected 401 on account not found") + assert.Equal(t, "Account 'testacc' doesn't support events", string(d)) +} + +func TestShouldNotPassEventToAnalyticsReporterWhenAccountEventNotEnabled(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=b&x=1&a=events_disabled", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 401, recorder.Result().StatusCode, "Expected 401 on account with events disabled") + assert.Equal(t, "Account 'events_disabled' doesn't support events", string(d)) +} + +func TestShouldPassEventToAnalyticsReporterWhenAccountEventEnabled(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=b&x=1&a=events_enabled", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + // validate + assert.Equal(t, 204, recorder.Result().StatusCode, "Expected 204 when account has events enabled") + assert.Equal(t, true, mockAnalyticsModule.Invoked) +} + +func TestShouldNotPassEventToAnalyticsReporterWhenAnalyticsValueIsZero(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=b&x=0&a=events_enabled", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + // validate + assert.Equal(t, 204, recorder.Result().StatusCode) + assert.Equal(t, true, mockAnalyticsModule.Invoked != true) +} + +func TestShouldRespondWithPixelAndContentTypeWhenRequestFormatIsImage(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=win&b=test&ts=1234&f=i&x=1&a=events_enabled", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 200, recorder.Result().StatusCode, "Expected 200 with tracking pixel when format is imp") + assert.Equal(t, true, mockAnalyticsModule.Invoked) + assert.Equal(t, "image/png", recorder.Header().Get("Content-Type")) + assert.Equal(t, "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAABHNCSVQICAgIfAhkiAAAAA1JREFUCJljYGBgYAAAAAUAAYehTtQAAAAASUVORK5CYII=", base64.URLEncoding.EncodeToString(d)) +} + +func TestShouldRespondWithNoContentWhenRequestFormatIsNotDefined(t *testing.T) { + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // mock PBS Analytics Module + mockAnalyticsModule := &eventsMockAnalyticsModule{ + Fail: false, + } + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("GET", "/event?t=imp&b=test&ts=1234&x=1&a=events_enabled", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := NewEventEndpoint(cfg, mockAccountsFetcher, mockAnalyticsModule) + + // execute + e(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 204, recorder.Result().StatusCode, "Expected 200 with empty response") + assert.Equal(t, true, mockAnalyticsModule.Invoked) + assert.Equal(t, "", recorder.Header().Get("Content-Type")) + assert.Equal(t, 0, len(d)) +} + +func TestShouldParseEventCorrectly(t *testing.T) { + + tests := map[string]struct { + req *http.Request + expected *analytics.EventRequest + }{ + "one": { + req: httptest.NewRequest("GET", "/event?t=win&b=bidId&f=b&ts=1000&x=1&a=accountId&bidder=bidder", strings.NewReader("")), + expected: &analytics.EventRequest{ + Type: analytics.Win, + BidID: "bidId", + Timestamp: 1000, + Bidder: "bidder", + AccountID: "", + Format: analytics.Blank, + Analytics: analytics.Enabled, + }, + }, + "two": { + req: httptest.NewRequest("GET", "/event?t=win&b=bidId&ts=0&a=accountId", strings.NewReader("")), + expected: &analytics.EventRequest{ + Type: analytics.Win, + BidID: "bidId", + Timestamp: 0, + Analytics: analytics.Enabled, + }, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + + // execute + er, errs := ParseEventRequest(test.req) + + // validate + assert.Equal(t, 0, len(errs)) + assert.EqualValues(t, test.expected, er) + }) + } +} + +func TestEventRequestToUrl(t *testing.T) { + externalUrl := "http://localhost:8000" + tests := map[string]struct { + er *analytics.EventRequest + want string + }{ + "one": { + er: &analytics.EventRequest{ + Type: analytics.Imp, + BidID: "bidid", + AccountID: "accountId", + Bidder: "bidder", + Timestamp: 1234567, + Format: analytics.Blank, + Analytics: analytics.Enabled, + }, + want: "http://localhost:8000/event?t=imp&b=bidid&a=accountId&bidder=bidder&f=b&ts=1234567&x=1", + }, + "two": { + er: &analytics.EventRequest{ + Type: analytics.Win, + BidID: "bidid", + AccountID: "accountId", + Bidder: "bidder", + Timestamp: 1234567, + Format: analytics.Image, + Analytics: analytics.Disabled, + }, + want: "http://localhost:8000/event?t=win&b=bidid&a=accountId&bidder=bidder&f=i&ts=1234567&x=0", + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + expected := EventRequestToUrl(externalUrl, test.er) + // validate + assert.Equal(t, test.want, expected) + }) + } +} diff --git a/endpoints/events/vtrack.go b/endpoints/events/vtrack.go new file mode 100644 index 00000000000..8a86e68edf1 --- /dev/null +++ b/endpoints/events/vtrack.go @@ -0,0 +1,300 @@ +package events + +import ( + "context" + "encoding/json" + "fmt" + accountService "github.com/PubMatic-OpenWrap/prebid-server/account" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/analytics" + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/golang/glog" + "github.com/julienschmidt/httprouter" + "io" + "io/ioutil" + "net/http" + "sort" + "strings" + "time" +) + +const ( + AccountParameter = "a" + ImpressionCloseTag = "" + ImpressionOpenTag = "" +) + +type vtrackEndpoint struct { + Cfg *config.Configuration + Accounts stored_requests.AccountFetcher + BidderInfos adapters.BidderInfos + Cache prebid_cache_client.Client +} + +type BidCacheRequest struct { + Puts []prebid_cache_client.Cacheable `json:"puts"` +} + +type BidCacheResponse struct { + Responses []CacheObject `json:"responses"` +} + +type CacheObject struct { + UUID string `json:"uuid"` +} + +func NewVTrackEndpoint(cfg *config.Configuration, accounts stored_requests.AccountFetcher, cache prebid_cache_client.Client, bidderInfos adapters.BidderInfos) httprouter.Handle { + vte := &vtrackEndpoint{ + Cfg: cfg, + Accounts: accounts, + BidderInfos: bidderInfos, + Cache: cache, + } + + return vte.Handle +} + +// /vtrack Handler +func (v *vtrackEndpoint) Handle(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { + + // get account id from request parameter + accountId := getAccountId(r) + + // account id is required + if accountId == "" { + w.WriteHeader(http.StatusBadRequest) + w.Write([]byte(fmt.Sprintf("Account '%s' is required query parameter and can't be empty", AccountParameter))) + return + } + + // parse puts request from request body + req, err := ParseVTrackRequest(r, v.Cfg.MaxRequestSize+1) + + // check if there was any error while parsing puts request + if err != nil { + w.WriteHeader(http.StatusBadRequest) + w.Write([]byte(fmt.Sprintf("Invalid request: %s\n", err.Error()))) + return + } + + ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(time.Duration(v.Cfg.VTrack.TimeoutMS)*time.Millisecond)) + defer cancel() + + // get account details + account, errs := accountService.GetAccount(ctx, v.Cfg, v.Accounts, accountId) + if len(errs) > 0 { + status, messages := HandleAccountServiceErrors(errs) + w.WriteHeader(status) + + for _, message := range messages { + w.Write([]byte(fmt.Sprintf("Invalid request: %s\n", message))) + } + return + } + + // insert impression tracking if account allows events and bidder allows VAST modification + if v.Cache != nil { + cachingResponse, errs := v.handleVTrackRequest(ctx, req, account) + + if len(errs) > 0 { + w.WriteHeader(http.StatusInternalServerError) + for _, err := range errs { + w.Write([]byte(fmt.Sprintf("Error(s) updating vast: %s\n", err.Error()))) + + return + } + } + + d, err := json.Marshal(*cachingResponse) + + if err != nil { + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(fmt.Sprintf("Error serializing pbs cache response: %s\n", err.Error()))) + + return + } + + w.Header().Add("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + w.Write(d) + + return + } + + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte("PBS Cache client is not configured")) +} + +// GetVastUrlTracking creates a vast url tracking +func GetVastUrlTracking(externalUrl string, bidid string, bidder string, accountId string, timestamp int64) string { + + eventReq := &analytics.EventRequest{ + Type: analytics.Imp, + BidID: bidid, + AccountID: accountId, + Bidder: bidder, + Timestamp: timestamp, + Format: analytics.Blank, + } + + return EventRequestToUrl(externalUrl, eventReq) +} + +// ParseVTrackRequest parses a BidCacheRequest from an HTTP Request +func ParseVTrackRequest(httpRequest *http.Request, maxRequestSize int64) (req *BidCacheRequest, err error) { + req = &BidCacheRequest{} + err = nil + + // Pull the request body into a buffer, so we have it for later usage. + lr := &io.LimitedReader{ + R: httpRequest.Body, + N: maxRequestSize, + } + + defer httpRequest.Body.Close() + requestJson, err := ioutil.ReadAll(lr) + if err != nil { + return req, err + } + + // Check if the request size was too large + if lr.N <= 0 { + err = &errortypes.BadInput{Message: fmt.Sprintf("request size exceeded max size of %d bytes", maxRequestSize-1)} + return req, err + } + + if len(requestJson) == 0 { + err = &errortypes.BadInput{Message: "request body is empty"} + return req, err + } + + if err := json.Unmarshal(requestJson, req); err != nil { + return req, err + } + + for _, bcr := range req.Puts { + if bcr.BidID == "" { + err = error(&errortypes.BadInput{Message: fmt.Sprint("'bidid' is required field and can't be empty")}) + return req, err + } + + if bcr.Bidder == "" { + err = error(&errortypes.BadInput{Message: fmt.Sprint("'bidder' is required field and can't be empty")}) + return req, err + } + } + + return req, nil +} + +// handleVTrackRequest handles a VTrack request +func (v *vtrackEndpoint) handleVTrackRequest(ctx context.Context, req *BidCacheRequest, account *config.Account) (*BidCacheResponse, []error) { + biddersAllowingVastUpdate := getBiddersAllowingVastUpdate(req, &v.BidderInfos, v.Cfg.VTrack.AllowUnknownBidder) + // cache data + r, errs := v.cachePutObjects(ctx, req, biddersAllowingVastUpdate, account.ID) + + // handle pbs caching errors + if len(errs) != 0 { + glog.Errorf("Error(s) updating vast: %v", errs) + return nil, errs + } + + // build response + response := &BidCacheResponse{ + Responses: []CacheObject{}, + } + + for _, uuid := range r { + response.Responses = append(response.Responses, CacheObject{ + UUID: uuid, + }) + } + + return response, nil +} + +// cachePutObjects caches BidCacheRequest data +func (v *vtrackEndpoint) cachePutObjects(ctx context.Context, req *BidCacheRequest, biddersAllowingVastUpdate map[string]struct{}, accountId string) ([]string, []error) { + var cacheables []prebid_cache_client.Cacheable + + for _, c := range req.Puts { + + nc := &prebid_cache_client.Cacheable{ + Type: c.Type, + Data: c.Data, + TTLSeconds: c.TTLSeconds, + Key: c.Key, + } + + if _, ok := biddersAllowingVastUpdate[c.Bidder]; ok && nc.Data != nil { + nc.Data = modifyVastXml(v.Cfg.ExternalURL, nc.Data, c.BidID, c.Bidder, accountId, c.Timestamp) + } + + cacheables = append(cacheables, *nc) + } + + return v.Cache.PutJson(ctx, cacheables) +} + +// getBiddersAllowingVastUpdate returns a list of bidders that allow VAST XML modification +func getBiddersAllowingVastUpdate(req *BidCacheRequest, bidderInfos *adapters.BidderInfos, allowUnknownBidder bool) map[string]struct{} { + bl := map[string]struct{}{} + + for _, bcr := range req.Puts { + if _, ok := bl[bcr.Bidder]; isAllowVastForBidder(bcr.Bidder, bidderInfos, allowUnknownBidder) && !ok { + bl[bcr.Bidder] = struct{}{} + } + } + + return bl +} + +// isAllowVastForBidder checks if a bidder is active and allowed to modify vast xml data +func isAllowVastForBidder(bidder string, bidderInfos *adapters.BidderInfos, allowUnknownBidder bool) bool { + //if bidder is active and isModifyingVastXmlAllowed is true + // check if bidder is configured + if b, ok := (*bidderInfos)[bidder]; bidderInfos != nil && ok { + // check if bidder is enabled + return b.Status == adapters.StatusActive && b.ModifyingVastXmlAllowed + } + + return allowUnknownBidder +} + +// getAccountId extracts an account id from an HTTP Request +func getAccountId(httpRequest *http.Request) string { + return httpRequest.URL.Query().Get(AccountParameter) +} + +// modifyVastXml modifies BidCacheRequest element Vast XML data +func modifyVastXml(externalUrl string, data json.RawMessage, bidid string, bidder string, accountId string, timestamp int64) json.RawMessage { + c := string(data) + ci := strings.Index(c, ImpressionCloseTag) + + // no impression tag - pass it as it is + if ci == -1 { + return data + } + + vastUrlTracking := GetVastUrlTracking(externalUrl, bidid, bidder, accountId, timestamp) + impressionUrl := "" + oi := strings.Index(c, ImpressionOpenTag) + + if ci-oi == len(ImpressionOpenTag) { + return json.RawMessage(strings.Replace(c, ImpressionOpenTag, ImpressionOpenTag+impressionUrl, 1)) + } + + return json.RawMessage(strings.Replace(c, ImpressionCloseTag, ImpressionCloseTag+ImpressionOpenTag+impressionUrl+ImpressionCloseTag, 1)) +} + +func contains(s []string, e string) bool { + if len(s) == 0 { + return false + } + + i := sort.SearchStrings(s, e) + return i < len(s) && s[i] == e +} diff --git a/endpoints/events/vtrack_test.go b/endpoints/events/vtrack_test.go new file mode 100644 index 00000000000..52665e7736d --- /dev/null +++ b/endpoints/events/vtrack_test.go @@ -0,0 +1,692 @@ +package events + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/stretchr/testify/assert" + "io/ioutil" + "net/http/httptest" + "strings" + "testing" +) + +const ( + maxSize = 1024 * 256 + + vastXmlWithImpressionWithContent = "prebid.org wrappercontent" + vastXmlWithImpressionWithoutContent = "prebid.org wrapper" + vastXmlWithoutImpression = "prebid.org wrapper" +) + +// Mock pbs cache client +type vtrackMockCacheClient struct { + Fail bool + Error error + Uuids []string +} + +func (m *vtrackMockCacheClient) PutJson(ctx context.Context, values []prebid_cache_client.Cacheable) ([]string, []error) { + if m.Fail { + return []string{}, []error{m.Error} + } + return m.Uuids, []error{} +} +func (m *vtrackMockCacheClient) GetExtCacheData() (scheme string, host string, path string) { + return +} + +// Test +func TestShouldRespondWithBadRequestWhenAccountParameterIsMissing(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{} + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // mock config + cfg := &config.Configuration{ + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("POST", "/vtrack", strings.NewReader(reqData)) + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with missing account parameter") + assert.Equal(t, "Account 'a' is required query parameter and can't be empty", string(d)) +} + +func TestShouldRespondWithBadRequestWhenRequestBodyIsEmpty(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{} + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "" + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(reqData)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with empty body") + assert.Equal(t, "Invalid request: request body is empty\n", string(d)) +} + +func TestShouldRespondWithBadRequestWhenRequestBodyIsInvalid(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{} + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + reqData := "invalid" + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(reqData)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with invalid body") +} + +func TestShouldRespondWithBadRequestWhenBidIdIsMissing(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{} + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + data := &BidCacheRequest{ + Puts: []prebid_cache_client.Cacheable{ + {}, + }, + } + + reqData, err := json.Marshal(data) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(string(reqData))) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with elements missing bidid") + assert.Equal(t, "Invalid request: 'bidid' is required field and can't be empty\n", string(d)) +} + +func TestShouldRespondWithBadRequestWhenBidderIsMissing(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{} + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + data := &BidCacheRequest{ + Puts: []prebid_cache_client.Cacheable{ + { + BidID: "test", + }, + }, + } + + reqData, err := json.Marshal(data) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(string(reqData))) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 on request with elements missing bidder") + assert.Equal(t, "Invalid request: 'bidder' is required field and can't be empty\n", string(d)) +} + +func TestShouldRespondWithInternalServerErrorWhenPbsCacheClientFails(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{ + Fail: true, + Error: fmt.Errorf("pbs cache client failed"), + } + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{} + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: true, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + data, err := getValidVTrackRequestBody(false, false) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(data)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 500, recorder.Result().StatusCode, "Expected 500 when pbs cache client fails") + assert.Equal(t, "Error(s) updating vast: pbs cache client failed\n", string(d)) +} + +func TestShouldTolerateAccountNotFound(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{} + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: true, + Error: stored_requests.NotFoundError{}, + } + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: false, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + data, err := getValidVTrackRequestBody(true, false) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=1235", strings.NewReader(data)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + // validate + assert.Equal(t, 200, recorder.Result().StatusCode, "Expected 200 when account is not found and request is valid") + assert.Equal(t, "application/json", recorder.Header().Get("Content-Type")) +} + +func TestShouldSendToCacheExpectedPutsAndUpdatableBiddersWhenBidderVastNotAllowed(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{ + Fail: false, + Uuids: []string{"uuid1"}, + } + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: false, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // bidder info + bidderInfos := make(adapters.BidderInfos) + bidderInfos["bidder"] = adapters.BidderInfo{ + Status: adapters.StatusActive, + ModifyingVastXmlAllowed: false, + } + bidderInfos["updatable_bidder"] = adapters.BidderInfo{ + Status: adapters.StatusActive, + ModifyingVastXmlAllowed: true, + } + + // prepare + data, err := getValidVTrackRequestBody(false, false) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(data)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: bidderInfos, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 200, recorder.Result().StatusCode, "Expected 200 when account is not found and request is valid") + assert.Equal(t, "{\"responses\":[{\"uuid\":\"uuid1\"}]}", string(d), "Expected 200 when account is found and request is valid") + assert.Equal(t, "application/json", recorder.Header().Get("Content-Type")) +} + +func TestShouldSendToCacheExpectedPutsAndUpdatableBiddersWhenBidderVastAllowed(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{ + Fail: false, + Uuids: []string{"uuid1", "uuid2"}, + } + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: false, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // bidder info + bidderInfos := make(adapters.BidderInfos) + bidderInfos["bidder"] = adapters.BidderInfo{ + Status: adapters.StatusActive, + ModifyingVastXmlAllowed: true, + } + bidderInfos["updatable_bidder"] = adapters.BidderInfo{ + Status: adapters.StatusActive, + ModifyingVastXmlAllowed: true, + } + + // prepare + data, err := getValidVTrackRequestBody(true, true) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(data)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: bidderInfos, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 200, recorder.Result().StatusCode, "Expected 200 when account is not found and request is valid") + assert.Equal(t, "{\"responses\":[{\"uuid\":\"uuid1\"},{\"uuid\":\"uuid2\"}]}", string(d), "Expected 200 when account is found and request is valid") + assert.Equal(t, "application/json", recorder.Header().Get("Content-Type")) +} + +func TestShouldSendToCacheExpectedPutsAndUpdatableUnknownBiddersWhenUnknownBidderIsAllowed(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{ + Fail: false, + Uuids: []string{"uuid1", "uuid2"}, + } + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: true, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // bidder info + bidderInfos := make(adapters.BidderInfos) + + // prepare + data, err := getValidVTrackRequestBody(true, false) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(data)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: bidderInfos, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 200, recorder.Result().StatusCode, "Expected 200 when account is not found and request is valid") + assert.Equal(t, "{\"responses\":[{\"uuid\":\"uuid1\"},{\"uuid\":\"uuid2\"}]}", string(d), "Expected 200 when account is found, request has unknown bidders but allowUnknownBidders is enabled") + assert.Equal(t, "application/json", recorder.Header().Get("Content-Type")) +} + +func TestShouldReturnBadRequestWhenRequestExceedsMaxRequestSize(t *testing.T) { + // mock pbs cache client + mockCacheClient := &vtrackMockCacheClient{ + Fail: false, + Uuids: []string{"uuid1", "uuid2"}, + } + + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // config + cfg := &config.Configuration{ + MaxRequestSize: 1, + VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: true, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // bidder info + bidderInfos := make(adapters.BidderInfos) + + // prepare + data, err := getValidVTrackRequestBody(true, false) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(data)) + + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: bidderInfos, + Cache: mockCacheClient, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 400, recorder.Result().StatusCode, "Expected 400 when request exceeds max request size") + assert.Equal(t, "Invalid request: request size exceeded max size of 1 bytes\n", string(d)) +} + +func TestShouldRespondWithInternalErrorPbsCacheIsNotConfigured(t *testing.T) { + // mock AccountsFetcher + mockAccountsFetcher := &mockAccountsFetcher{ + Fail: false, + } + + // config + cfg := &config.Configuration{ + MaxRequestSize: maxSize, VTrack: config.VTrack{ + TimeoutMS: int64(2000), AllowUnknownBidder: false, + }, + AccountDefaults: config.Account{}, + } + cfg.MarshalAccountDefaults() + + // prepare + data, err := getValidVTrackRequestBody(true, true) + if err != nil { + t.Fatal(err) + } + + req := httptest.NewRequest("POST", "/vtrack?a=events_enabled", strings.NewReader(data)) + recorder := httptest.NewRecorder() + + e := vtrackEndpoint{ + Cfg: cfg, + BidderInfos: nil, + Cache: nil, + Accounts: mockAccountsFetcher, + } + + // execute + e.Handle(recorder, req, nil) + + d, err := ioutil.ReadAll(recorder.Result().Body) + if err != nil { + t.Fatal(err) + } + + // validate + assert.Equal(t, 500, recorder.Result().StatusCode, "Expected 500 when pbs cache is not configured") + assert.Equal(t, "PBS Cache client is not configured", string(d)) +} + +func TestVastUrlShouldReturnExpectedUrl(t *testing.T) { + url := GetVastUrlTracking("http://external-url", "bidId", "bidder", "accountId", 1000) + assert.Equal(t, "http://external-url/event?t=imp&b=bidId&a=accountId&bidder=bidder&f=b&ts=1000", url, "Invalid vast url") +} + +func getValidVTrackRequestBody(withImpression bool, withContent bool) (string, error) { + d, e := getVTrackRequestData(withImpression, withContent) + + if e != nil { + return "", e + } + + req := &BidCacheRequest{ + Puts: []prebid_cache_client.Cacheable{ + { + Type: prebid_cache_client.TypeXML, + BidID: "bidId1", + Bidder: "bidder", + Data: d, + TTLSeconds: 3600, + Timestamp: 1000, + }, + { + Type: prebid_cache_client.TypeXML, + BidID: "bidId2", + Bidder: "updatable_bidder", + Data: d, + TTLSeconds: 3600, + Timestamp: 1000, + }, + }, + } + + buf := &bytes.Buffer{} + enc := json.NewEncoder(buf) + enc.SetEscapeHTML(false) + + e = enc.Encode(req) + + return buf.String(), e +} + +func getVTrackRequestData(wi bool, wic bool) (db []byte, e error) { + data := &bytes.Buffer{} + enc := json.NewEncoder(data) + enc.SetEscapeHTML(false) + + if wi && wic { + e = enc.Encode(vastXmlWithImpressionWithContent) + return data.Bytes(), e + } else if wi { + e = enc.Encode(vastXmlWithImpressionWithoutContent) + } else { + enc.Encode(vastXmlWithoutImpression) + } + + return data.Bytes(), e +} diff --git a/endpoints/openrtb2/amp_auction.go b/endpoints/openrtb2/amp_auction.go index cb36528417b..cd38e4c95ef 100644 --- a/endpoints/openrtb2/amp_auction.go +++ b/endpoints/openrtb2/amp_auction.go @@ -13,6 +13,7 @@ import ( "time" "github.com/PubMatic-OpenWrap/openrtb" + accountService "github.com/PubMatic-OpenWrap/prebid-server/account" "github.com/PubMatic-OpenWrap/prebid-server/analytics" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/errortypes" @@ -20,9 +21,12 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" "github.com/PubMatic-OpenWrap/prebid-server/privacy" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/empty_fetcher" "github.com/PubMatic-OpenWrap/prebid-server/usersync" + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" "github.com/buger/jsonparser" "github.com/golang/glog" "github.com/julienschmidt/httprouter" @@ -43,7 +47,7 @@ func NewAmpEndpoint( ex exchange.Exchange, validator openrtb_ext.BidderParamValidator, requestsById stored_requests.Fetcher, - categories stored_requests.CategoryFetcher, + accounts stored_requests.AccountFetcher, cfg *config.Configuration, met pbsmetrics.MetricsEngine, pbsAnalytics analytics.PBSAnalyticsModule, @@ -52,18 +56,23 @@ func NewAmpEndpoint( bidderMap map[string]openrtb_ext.BidderName, ) (httprouter.Handle, error) { - if ex == nil || validator == nil || requestsById == nil || cfg == nil || met == nil { + if ex == nil || validator == nil || requestsById == nil || accounts == nil || cfg == nil || met == nil { return nil, errors.New("NewAmpEndpoint requires non-nil arguments.") } defRequest := defReqJSON != nil && len(defReqJSON) > 0 + ipValidator := iputil.PublicNetworkIPValidator{ + IPv4PrivateNetworks: cfg.RequestValidation.IPv4PrivateNetworksParsed, + IPv6PrivateNetworks: cfg.RequestValidation.IPv6PrivateNetworksParsed, + } + return httprouter.Handle((&endpointDeps{ ex, validator, requestsById, empty_fetcher.EmptyFetcher{}, - categories, + accounts, cfg, met, pbsAnalytics, @@ -72,7 +81,8 @@ func NewAmpEndpoint( defReqJSON, bidderMap, nil, - nil}).AmpAuction), nil + nil, + ipValidator}).AmpAuction), nil } @@ -132,6 +142,8 @@ func (deps *endpointDeps) AmpAuction(w http.ResponseWriter, r *http.Request, _ h return } + ao.Request = req + ctx := context.Background() var cancel context.CancelFunc if req.TMax > 0 { @@ -147,26 +159,39 @@ func (deps *endpointDeps) AmpAuction(w http.ResponseWriter, r *http.Request, _ h } else { labels.CookieFlag = pbsmetrics.CookieFlagYes } - labels.PubID = effectivePubID(req.Site.Publisher) - // Blacklist account now that we have resolved the value - if acctIdErr := validateAccount(deps.cfg, labels.PubID); acctIdErr != nil { - errL = append(errL, acctIdErr) - errCode := errortypes.ReadCode(acctIdErr) - if errCode == errortypes.BlacklistedAppErrorCode || errCode == errortypes.BlacklistedAcctErrorCode { - w.WriteHeader(http.StatusServiceUnavailable) - labels.RequestStatus = pbsmetrics.RequestStatusBlacklisted - } else { - w.WriteHeader(http.StatusBadRequest) - labels.RequestStatus = pbsmetrics.RequestStatusBadInput + labels.PubID = getAccountID(req.Site.Publisher) + // Look up account now that we have resolved the pubID value + account, acctIDErrs := accountService.GetAccount(ctx, deps.cfg, deps.accounts, labels.PubID) + if len(acctIDErrs) > 0 { + errL = append(errL, acctIDErrs...) + httpStatus := http.StatusBadRequest + metricsStatus := pbsmetrics.RequestStatusBadInput + for _, er := range errL { + errCode := errortypes.ReadCode(er) + if errCode == errortypes.BlacklistedAppErrorCode || errCode == errortypes.BlacklistedAcctErrorCode { + httpStatus = http.StatusServiceUnavailable + metricsStatus = pbsmetrics.RequestStatusBlacklisted + break + } } + w.WriteHeader(httpStatus) + labels.RequestStatus = metricsStatus for _, err := range errortypes.FatalOnly(errL) { w.Write([]byte(fmt.Sprintf("Invalid request format: %s\n", err.Error()))) } - ao.Errors = append(ao.Errors, acctIdErr) + ao.Errors = append(ao.Errors, acctIDErrs...) return } - response, err := deps.ex.HoldAuction(ctx, req, usersyncs, labels, &deps.categories, nil) + auctionRequest := exchange.AuctionRequest{ + BidRequest: req, + Account: *account, + UserSyncs: usersyncs, + RequestType: labels.RType, + LegacyLabels: labels, + } + + response, err := deps.ex.HoldAuction(ctx, auctionRequest, nil) ao.AuctionResponse = response if err != nil { @@ -381,22 +406,19 @@ func (deps *endpointDeps) overrideWithParams(httpRequest *http.Request, req *ope setAmpExt(req.Site, "1") + setEffectiveAmpPubID(req, httpRequest.URL.Query()) + slot := httpRequest.FormValue("slot") if slot != "" { req.Imp[0].TagID = slot } - consent := readConsent(httpRequest.URL) - if consent != "" { - if policies, ok := privacy.ReadPoliciesFromConsent(consent); ok { - if err := policies.Write(req); err != nil { - return []error{err} - } - } else { - return []error{&errortypes.InvalidPrivacyConsent{ - Message: fmt.Sprintf("Consent '%s' is not recognized as either CCPA or GDPR TCF.", consent), - }} - } + policyWriter, policyWriterErr := readPolicyFromUrl(httpRequest.URL) + if policyWriterErr != nil { + return []error{policyWriterErr} + } + if err := policyWriter.Write(req); err != nil { + return []error{err} } if timeout, err := strconv.ParseInt(httpRequest.FormValue("timeout"), 10, 64); err == nil { @@ -541,7 +563,27 @@ func setAmpExt(site *openrtb.Site, value string) { } } -func readConsent(url *url.URL) string { +func readPolicyFromUrl(url *url.URL) (privacy.PolicyWriter, error) { + consent := readConsentFromURL(url) + + if len(consent) == 0 { + return privacy.NilPolicyWriter{}, nil + } + + if gdpr.ValidateConsent(consent) { + return gdpr.ConsentWriter{consent}, nil + } + + if ccpa.ValidateConsent(consent) { + return ccpa.ConsentWriter{consent}, nil + } + + return privacy.NilPolicyWriter{}, &errortypes.InvalidPrivacyConsent{ + Message: fmt.Sprintf("Consent '%s' is not recognized as either CCPA or GDPR TCF.", consent), + } +} + +func readConsentFromURL(url *url.URL) string { if v := url.Query().Get("consent_string"); v != "" { return v } @@ -549,3 +591,27 @@ func readConsent(url *url.URL) string { // Fallback to 'gdpr_consent' for compatability until it's no longer used by AMP. return url.Query().Get("gdpr_consent") } + +// Sets the effective publisher ID for amp request +func setEffectiveAmpPubID(req *openrtb.BidRequest, urlQueryParams url.Values) { + var pub *openrtb.Publisher + if req.App != nil { + if req.App.Publisher == nil { + req.App.Publisher = new(openrtb.Publisher) + } + pub = req.App.Publisher + } else if req.Site != nil { + if req.Site.Publisher == nil { + req.Site.Publisher = new(openrtb.Publisher) + } + pub = req.Site.Publisher + } + + if pub.ID == "" { + // For amp requests, the publisher ID could be sent via the account + // query string + if acc := urlQueryParams.Get("account"); acc != "" && acc != "ACCOUNT_ID" { + pub.ID = acc + } + } +} diff --git a/endpoints/openrtb2/amp_auction_test.go b/endpoints/openrtb2/amp_auction_test.go index 259992dbe20..3ec5d477c22 100644 --- a/endpoints/openrtb2/amp_auction_test.go +++ b/endpoints/openrtb2/amp_auction_test.go @@ -11,7 +11,7 @@ import ( "strconv" "testing" - "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/PubMatic-OpenWrap/prebid-server/analytics" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/empty_fetcher" "github.com/PubMatic-OpenWrap/openrtb" @@ -755,8 +755,9 @@ func TestQueryParamOverrides(t *testing.T) { curl := "http://example.com" slot := "1234" timeout := int64(500) + account := "12345" - request := httptest.NewRequest("GET", fmt.Sprintf("/openrtb2/auction/amp?tag_id=%s&debug=1&curl=%s&slot=%s&timeout=%d", requestID, curl, slot, timeout), nil) + request := httptest.NewRequest("GET", fmt.Sprintf("/openrtb2/auction/amp?tag_id=%s&debug=1&curl=%s&slot=%s&timeout=%d&account=%s", requestID, curl, slot, timeout, account), nil) recorder := httptest.NewRecorder() endpoint(recorder, request, nil) @@ -784,6 +785,10 @@ func TestQueryParamOverrides(t *testing.T) { if resolvedRequest.Site == nil || resolvedRequest.Site.Page != curl { t.Errorf("Expected Site.Page to equal curl (%s), got: %s", curl, resolvedRequest.Site.Page) } + + if resolvedRequest.Site == nil || resolvedRequest.Site.Publisher == nil || resolvedRequest.Site.Publisher.ID != account { + t.Errorf("Expected Site.Publisher.ID to equal (%s), got: %s", account, resolvedRequest.Site.Publisher.ID) + } } func TestOverrideDimensions(t *testing.T) { @@ -876,6 +881,7 @@ type formatOverrideSpec struct { overrideWidth uint64 overrideHeight uint64 multisize string + account string expect []openrtb.Format } @@ -897,7 +903,7 @@ func (s formatOverrideSpec) execute(t *testing.T) { openrtb_ext.BidderMap, ) - url := fmt.Sprintf("/openrtb2/auction/amp?tag_id=1&debug=1&w=%d&h=%d&ow=%d&oh=%d&ms=%s", s.width, s.height, s.overrideWidth, s.overrideHeight, s.multisize) + url := fmt.Sprintf("/openrtb2/auction/amp?tag_id=1&debug=1&w=%d&h=%d&ow=%d&oh=%d&ms=%s&account=%s", s.width, s.height, s.overrideWidth, s.overrideHeight, s.multisize, s.account) request := httptest.NewRequest("GET", url, nil) recorder := httptest.NewRecorder() endpoint(recorder, request, nil) @@ -946,8 +952,8 @@ var expectedErrorsFromHoldAuction map[openrtb_ext.BidderName][]openrtb_ext.ExtBi }, } -func (m *mockAmpExchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, ids exchange.IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { - m.lastRequest = bidRequest +func (m *mockAmpExchange) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + m.lastRequest = r.BidRequest response := &openrtb.BidResponse{ SeatBid: []openrtb.SeatBid{{ @@ -959,8 +965,8 @@ func (m *mockAmpExchange) HoldAuction(ctx context.Context, bidRequest *openrtb.B Ext: json.RawMessage(`{ "errors": {"openx":[ { "code": 1, "message": "The request exceeded the timeout allocated" } ] } }`), } - if bidRequest.Test == 1 { - resolvedRequest, err := json.Marshal(bidRequest) + if r.BidRequest.Test == 1 { + resolvedRequest, err := json.Marshal(r.BidRequest) if err != nil { resolvedRequest = json.RawMessage("{}") } @@ -1036,3 +1042,243 @@ func getTestBidRequest(nilUser bool, userExt *openrtb_ext.ExtUser, nilRegs bool, return json.Marshal(bidRequest) } + +func TestSetEffectiveAmpPubID(t *testing.T) { + testPubID := "test-pub" + testURLQueryParams := url.Values{} + testURLQueryParams.Add("account", testPubID) + + testCases := []struct { + description string + req *openrtb.BidRequest + urlQueryParams url.Values + expectedPubID string + }{ + { + description: "No publisher ID provided", + req: &openrtb.BidRequest{ + App: &openrtb.App{ + Publisher: nil, + }, + }, + expectedPubID: "", + }, + { + description: "Publisher ID present in req.App.Publisher.ID", + req: &openrtb.BidRequest{ + App: &openrtb.App{ + Publisher: &openrtb.Publisher{ + ID: testPubID, + }, + }, + }, + expectedPubID: testPubID, + }, + { + description: "Publisher ID present in req.Site.Publisher.ID", + req: &openrtb.BidRequest{ + Site: &openrtb.Site{ + Publisher: &openrtb.Publisher{ + ID: testPubID, + }, + }, + }, + expectedPubID: testPubID, + }, + { + description: "Publisher ID present in account query parameter", + req: &openrtb.BidRequest{ + App: &openrtb.App{ + Publisher: &openrtb.Publisher{ + ID: "", + }, + }, + }, + urlQueryParams: testURLQueryParams, + expectedPubID: testPubID, + }, + { + description: "req.Site.Publisher present but ID set to empty string", + req: &openrtb.BidRequest{ + Site: &openrtb.Site{ + Publisher: &openrtb.Publisher{ + ID: "", + }, + }, + }, + expectedPubID: "", + }, + } + + for _, test := range testCases { + setEffectiveAmpPubID(test.req, test.urlQueryParams) + if test.req.Site != nil { + assert.Equal(t, test.expectedPubID, test.req.Site.Publisher.ID, + "should return the expected Publisher ID for test case: %s", test.description) + } else { + assert.Equal(t, test.expectedPubID, test.req.App.Publisher.ID, + "should return the expected Publisher ID for test case: %s", test.description) + } + } +} + +type mockLogger struct { + ampObject *analytics.AmpObject +} + +func newMockLogger(ao *analytics.AmpObject) analytics.PBSAnalyticsModule { + return &mockLogger{ + ampObject: ao, + } +} + +func (logger mockLogger) LogAuctionObject(ao *analytics.AuctionObject) { + return +} +func (logger mockLogger) LogVideoObject(vo *analytics.VideoObject) { + return +} +func (logger mockLogger) LogCookieSyncObject(cookieObject *analytics.CookieSyncObject) { + return +} +func (logger mockLogger) LogSetUIDObject(uuidObj *analytics.SetUIDObject) { + return +} +func (logger mockLogger) LogNotificationEventObject(uuidObj *analytics.NotificationEvent) { + return +} +func (logger mockLogger) LogAmpObject(ao *analytics.AmpObject) { + *logger.ampObject = *ao +} + +func TestBuildAmpObject(t *testing.T) { + testCases := []struct { + description string + inTagId string + inStoredRequest json.RawMessage + expectedAmpObject *analytics.AmpObject + }{ + { + description: "Stored Amp request with nil body. Only the error gets logged", + inTagId: "test", + inStoredRequest: nil, + expectedAmpObject: &analytics.AmpObject{ + Status: http.StatusOK, + Errors: []error{fmt.Errorf("unexpected end of JSON input")}, + }, + }, + { + description: "Stored Amp request with no imps that should return error. Only the error gets logged", + inTagId: "test", + inStoredRequest: json.RawMessage(`{"id":"some-request-id","site":{"page":"prebid.org"},"imp":[],"tmax":500}`), + expectedAmpObject: &analytics.AmpObject{ + Status: http.StatusOK, + Errors: []error{fmt.Errorf("data for tag_id='test' does not define the required imp array")}, + }, + }, + { + description: "Wrong tag_id, error gets logged", + inTagId: "unknown", + inStoredRequest: json.RawMessage(`{"id":"some-request-id","site":{"page":"prebid.org"},"imp":[{"id":"some-impression-id","banner":{"format":[{"w":300,"h":250}]},"ext":{"appnexus":{"placementId":12883451}}}],"tmax":500}`), + expectedAmpObject: &analytics.AmpObject{ + Status: http.StatusOK, + Errors: []error{fmt.Errorf("unexpected end of JSON input")}, + }, + }, + { + description: "Valid stored Amp request, correct tag_id, a valid response should be logged", + inTagId: "test", + inStoredRequest: json.RawMessage(`{"id":"some-request-id","site":{"page":"prebid.org"},"imp":[{"id":"some-impression-id","banner":{"format":[{"w":300,"h":250}]},"ext":{"appnexus":{"placementId":12883451}}}],"tmax":500}`), + expectedAmpObject: &analytics.AmpObject{ + Status: http.StatusOK, + Errors: nil, + Request: &openrtb.BidRequest{ + ID: "some-request-id", + Device: &openrtb.Device{ + IP: "192.0.2.1", + }, + Site: &openrtb.Site{ + Page: "prebid.org", + Publisher: &openrtb.Publisher{}, + Ext: json.RawMessage(`{"amp":1}`), + }, + Imp: []openrtb.Imp{ + { + ID: "some-impression-id", + Banner: &openrtb.Banner{ + Format: []openrtb.Format{ + { + W: 300, + H: 250, + }, + }, + }, + Secure: func(val int8) *int8 { return &val }(1), //(*int8)(1), + Ext: json.RawMessage(`{"appnexus":{"placementId":12883451}}`), + }, + }, + AT: 1, + TMax: 500, + Ext: json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":null},"vastxml":null},"targeting":{"pricegranularity":{"precision":2,"ranges":[{"min":0,"max":20,"increment":0.1}]},"includewinners":true,"includebidderkeys":true,"includebrandcategory":null,"includeformat":false,"durationrangesec":null,"preferdeals":false}}}`), + }, + AuctionResponse: &openrtb.BidResponse{ + SeatBid: []openrtb.SeatBid{{ + Bid: []openrtb.Bid{{ + AdM: "", + Ext: json.RawMessage(`{ "prebid": {"targeting": { "hb_pb": "1.20", "hb_appnexus_pb": "1.20", "hb_cache_id": "some_id"}}}`), + }}, + Seat: "", + }}, + Ext: json.RawMessage(`{ "errors": {"openx":[ { "code": 1, "message": "The request exceeded the timeout allocated" } ] } }`), + }, + AmpTargetingValues: map[string]string{ + "hb_appnexus_pb": "1.20", + "hb_cache_id": "some_id", + "hb_pb": "1.20", + }, + Origin: "", + }, + }, + } + + request := httptest.NewRequest("GET", "/openrtb2/auction/amp?tag_id=test", nil) + recorder := httptest.NewRecorder() + + for _, test := range testCases { + + // Set up test, declare a new mock logger every time + actualAmpObject := new(analytics.AmpObject) + + logger := newMockLogger(actualAmpObject) + + mockAmpFetcher := &mockAmpStoredReqFetcher{ + data: map[string]json.RawMessage{ + test.inTagId: json.RawMessage(test.inStoredRequest), + }, + } + + endpoint, _ := NewAmpEndpoint( + &mockAmpExchange{}, + newParamsValidator(t), + mockAmpFetcher, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{MaxRequestSize: maxSize}, + pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), + logger, + map[string]string{}, + []byte{}, + openrtb_ext.BidderMap, + ) + + // Run test + endpoint(recorder, request, nil) + + // assert AmpObject + assert.Equalf(t, test.expectedAmpObject.Status, actualAmpObject.Status, "Amp Object Status field doesn't match expected: %s\n", test.description) + assert.Lenf(t, actualAmpObject.Errors, len(test.expectedAmpObject.Errors), "Amp Object Errors array doesn't match expected: %s\n", test.description) + assert.Equalf(t, test.expectedAmpObject.Request, actualAmpObject.Request, "Amp Object BidRequest doesn't match expected: %s\n", test.description) + assert.Equalf(t, test.expectedAmpObject.AuctionResponse, actualAmpObject.AuctionResponse, "Amp Object BidResponse doesn't match expected: %s\n", test.description) + assert.Equalf(t, test.expectedAmpObject.AmpTargetingValues, actualAmpObject.AmpTargetingValues, "Amp Object AmpTargetingValues doesn't match expected: %s\n", test.description) + assert.Equalf(t, test.expectedAmpObject.Origin, actualAmpObject.Origin, "Amp Object Origin field doesn't match expected: %s\n", test.description) + } +} diff --git a/endpoints/openrtb2/auction.go b/endpoints/openrtb2/auction.go index f8552666bc3..73bfa410441 100644 --- a/endpoints/openrtb2/auction.go +++ b/endpoints/openrtb2/auction.go @@ -16,20 +16,23 @@ import ( "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/openrtb/native" nativeRequests "github.com/PubMatic-OpenWrap/openrtb/native/request" + accountService "github.com/PubMatic-OpenWrap/prebid-server/account" "github.com/PubMatic-OpenWrap/prebid-server/analytics" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/errortypes" "github.com/PubMatic-OpenWrap/prebid-server/exchange" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" - "github.com/PubMatic-OpenWrap/prebid-server/prebid" "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/empty_fetcher" "github.com/PubMatic-OpenWrap/prebid-server/usersync" + "github.com/PubMatic-OpenWrap/prebid-server/util/httputil" + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" "github.com/buger/jsonparser" jsonpatch "github.com/evanphx/json-patch" + "github.com/gofrs/uuid" "github.com/golang/glog" "github.com/julienschmidt/httprouter" "github.com/mssola/user_agent" @@ -38,19 +41,31 @@ import ( const storedRequestTimeoutMillis = 50 -func NewEndpoint(ex exchange.Exchange, validator openrtb_ext.BidderParamValidator, requestsById stored_requests.Fetcher, categories stored_requests.CategoryFetcher, cfg *config.Configuration, met pbsmetrics.MetricsEngine, pbsAnalytics analytics.PBSAnalyticsModule, disabledBidders map[string]string, defReqJSON []byte, bidderMap map[string]openrtb_ext.BidderName) (httprouter.Handle, error) { +var ( + dntKey string = http.CanonicalHeaderKey("DNT") + dntDisabled int8 = 0 + dntEnabled int8 = 1 +) + +func NewEndpoint(ex exchange.Exchange, validator openrtb_ext.BidderParamValidator, requestsById stored_requests.Fetcher, accounts stored_requests.AccountFetcher, cfg *config.Configuration, met pbsmetrics.MetricsEngine, pbsAnalytics analytics.PBSAnalyticsModule, disabledBidders map[string]string, defReqJSON []byte, bidderMap map[string]openrtb_ext.BidderName) (httprouter.Handle, error) { - if ex == nil || validator == nil || requestsById == nil || cfg == nil || met == nil { + if ex == nil || validator == nil || requestsById == nil || accounts == nil || cfg == nil || met == nil { return nil, errors.New("NewEndpoint requires non-nil arguments.") } + defRequest := defReqJSON != nil && len(defReqJSON) > 0 + ipValidator := iputil.PublicNetworkIPValidator{ + IPv4PrivateNetworks: cfg.RequestValidation.IPv4PrivateNetworksParsed, + IPv6PrivateNetworks: cfg.RequestValidation.IPv6PrivateNetworksParsed, + } + return httprouter.Handle((&endpointDeps{ ex, validator, requestsById, empty_fetcher.EmptyFetcher{}, - categories, + accounts, cfg, met, pbsAnalytics, @@ -59,24 +74,26 @@ func NewEndpoint(ex exchange.Exchange, validator openrtb_ext.BidderParamValidato defReqJSON, bidderMap, nil, - nil}).Auction), nil + nil, + ipValidator}).Auction), nil } type endpointDeps struct { - ex exchange.Exchange - paramsValidator openrtb_ext.BidderParamValidator - storedReqFetcher stored_requests.Fetcher - videoFetcher stored_requests.Fetcher - categories stored_requests.CategoryFetcher - cfg *config.Configuration - metricsEngine pbsmetrics.MetricsEngine - analytics analytics.PBSAnalyticsModule - disabledBidders map[string]string - defaultRequest bool - defReqJSON []byte - bidderMap map[string]openrtb_ext.BidderName - cache prebid_cache_client.Client - debugLogRegexp *regexp.Regexp + ex exchange.Exchange + paramsValidator openrtb_ext.BidderParamValidator + storedReqFetcher stored_requests.Fetcher + videoFetcher stored_requests.Fetcher + accounts stored_requests.AccountFetcher + cfg *config.Configuration + metricsEngine pbsmetrics.MetricsEngine + analytics analytics.PBSAnalyticsModule + disabledBidders map[string]string + defaultRequest bool + defReqJSON []byte + bidderMap map[string]openrtb_ext.BidderName + cache prebid_cache_client.Client + debugLogRegexp *regexp.Regexp + privateNetworkIPValidator iputil.IPValidator } func (deps *endpointDeps) Auction(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { @@ -126,7 +143,7 @@ func (deps *endpointDeps) Auction(w http.ResponseWriter, r *http.Request, _ http if req.App != nil { labels.Source = pbsmetrics.DemandApp labels.RType = pbsmetrics.ReqTypeORTB2App - labels.PubID = effectivePubID(req.App.Publisher) + labels.PubID = getAccountID(req.App.Publisher) } else { //req.Site != nil labels.Source = pbsmetrics.DemandWeb if usersyncs.LiveSyncCount() == 0 { @@ -134,18 +151,29 @@ func (deps *endpointDeps) Auction(w http.ResponseWriter, r *http.Request, _ http } else { labels.CookieFlag = pbsmetrics.CookieFlagYes } - labels.PubID = effectivePubID(req.Site.Publisher) + labels.PubID = getAccountID(req.Site.Publisher) } - if acctIdErr := validateAccount(deps.cfg, labels.PubID); acctIdErr != nil { - errL = append(errL, acctIdErr) + // Look up account now that we have resolved the pubID value + account, acctIDErrs := accountService.GetAccount(ctx, deps.cfg, deps.accounts, labels.PubID) + if len(acctIDErrs) > 0 { + errL = append(errL, acctIDErrs...) writeError(errL, w, &labels) return } - response, err := deps.ex.HoldAuction(ctx, req, usersyncs, labels, &deps.categories, nil) + auctionRequest := exchange.AuctionRequest{ + BidRequest: req, + Account: *account, + UserSyncs: usersyncs, + RequestType: labels.RType, + LegacyLabels: labels, + } + + response, err := deps.ex.HoldAuction(ctx, auctionRequest, nil) ao.Request = req ao.Response = response + ao.Account = account if err != nil { labels.RequestStatus = pbsmetrics.RequestStatusErr w.WriteHeader(http.StatusInternalServerError) @@ -268,6 +296,14 @@ func (deps *endpointDeps) validateRequest(req *openrtb.BidRequest) []error { errL = append(errL, &errortypes.Warning{Message: fmt.Sprintf("A prebid request can only process one currency. Taking the first currency in the list, %s, as the active currency", req.Cur[0])}) } + // If automatically filling source TID is enabled then validate that + // source.TID exists and If it doesn't, fill it with a randomly generated UUID + if deps.cfg.AutoGenSourceTID { + if err := validateAndFillSourceTID(req); err != nil { + return []error{err} + } + } + var aliases map[string]string if bidExt, err := deps.parseBidExt(req.Ext); err != nil { return []error{err} @@ -281,45 +317,43 @@ func (deps *endpointDeps) validateRequest(req *openrtb.BidRequest) []error { if err := validateBidAdjustmentFactors(bidExt.Prebid.BidAdjustmentFactors, aliases); err != nil { return []error{err} } + + if err := validateSChains(bidExt); err != nil { + return []error{err} + } } if (req.Site == nil && req.App == nil) || (req.Site != nil && req.App != nil) { - errL = append(errL, errors.New("request.site or request.app must be defined, but not both.")) - return errL + return append(errL, errors.New("request.site or request.app must be defined, but not both.")) } if err := deps.validateSite(req.Site); err != nil { - errL = append(errL, err) - return errL + return append(errL, err) } if err := deps.validateApp(req.App); err != nil { - errL = append(errL, err) - return errL + return append(errL, err) } if err := validateUser(req.User, aliases); err != nil { - errL = append(errL, err) - return errL + return append(errL, err) } if err := validateRegs(req.Regs); err != nil { - errL = append(errL, err) - return errL - } - - ccpaPolicy, ccpaPolicyErr := ccpa.ReadPolicy(req) - if ccpaPolicyErr != nil { - errL = append(errL, ccpaPolicyErr) - return errL + return append(errL, err) } - if err := ccpaPolicy.Validate(); err != nil { - errL = append(errL, &errortypes.InvalidPrivacyConsent{Message: fmt.Sprintf("CCPA consent is invalid and will be ignored. (%v)", err)}) - - ccpaPolicy.Value = "" - if err := ccpaPolicy.Write(req); err != nil { - errL = append(errL, fmt.Errorf("Unable to remove invalid CCPA consent from the request. (%v)", err)) + if ccpaPolicy, err := ccpa.ReadFromRequest(req); err != nil { + return append(errL, err) + } else if _, err := ccpaPolicy.Parse(exchange.GetValidBidders(aliases)); err != nil { + if _, invalidConsent := err.(*errortypes.InvalidPrivacyConsent); invalidConsent { + errL = append(errL, &errortypes.InvalidPrivacyConsent{Message: fmt.Sprintf("CCPA consent is invalid and will be ignored. (%v)", err)}) + consentWriter := ccpa.ConsentWriter{""} + if err := consentWriter.Write(req); err != nil { + return append(errL, fmt.Errorf("Unable to remove invalid CCPA consent from the request. (%v)", err)) + } + } else { + return append(errL, err) } } @@ -342,6 +376,20 @@ func (deps *endpointDeps) validateRequest(req *openrtb.BidRequest) []error { return errL } +func validateAndFillSourceTID(req *openrtb.BidRequest) error { + if req.Source == nil { + req.Source = &openrtb.Source{} + } + if req.Source.TID == "" { + if rawUUID, err := uuid.NewV4(); err == nil { + req.Source.TID = rawUUID.String() + } else { + return errors.New("req.Source.TID missing in the req and error creating a random UID") + } + } + return nil +} + func validateBidAdjustmentFactors(adjustmentFactors map[string]float64, aliases map[string]string) error { for bidderToAdjust, adjustmentFactor := range adjustmentFactors { if adjustmentFactor <= 0 { @@ -356,6 +404,11 @@ func validateBidAdjustmentFactors(adjustmentFactors map[string]float64, aliases return nil } +func validateSChains(req *openrtb_ext.ExtRequest) error { + _, err := exchange.BidderToPrebidSChains(req) + return err +} + func (deps *endpointDeps) validateImp(imp *openrtb.Imp, aliases map[string]string, index int) []error { if imp.ID == "" { return []error{fmt.Errorf("request.imp[%d] missing required field: \"id\"", index)} @@ -716,8 +769,8 @@ func (deps *endpointDeps) validateImpExt(imp *openrtb.Imp, aliases map[string]st } // Also accept bidder exts within imp[...].ext.prebid.bidder - // NOTE: This is not part of the official API, we are not expecting clients - // migrate from imp[...].ext.${BIDDER} to imp[...].ext.prebid.bidder.${BIDDER} + // NOTE: This is not part of the official API yet, so we are not expecting clients + // to migrate from imp[...].ext.${BIDDER} to imp[...].ext.prebid.bidder.${BIDDER} // at this time // https://github.com/PubMatic-OpenWrap/prebid-server/pull/846#issuecomment-476352224 if rawPrebidExt, ok := bidderExts[openrtb_ext.PrebidExtKey]; ok { @@ -736,8 +789,9 @@ func (deps *endpointDeps) validateImpExt(imp *openrtb.Imp, aliases map[string]st /* Process all the bidder exts in the request */ disabledBidders := []string{} validationFailedBidders := []string{} + otherExtElements := 0 for bidder, ext := range bidderExts { - if bidder != openrtb_ext.PrebidExtKey { + if isBidderToValidate(bidder) { coreBidder := bidder if tmp, isAlias := aliases[bidder]; isAlias { coreBidder = tmp @@ -757,6 +811,8 @@ func (deps *endpointDeps) validateImpExt(imp *openrtb.Imp, aliases map[string]st return []error{fmt.Errorf("request.imp[%d].ext contains unknown bidder: %s. Did you forget an alias in request.ext.prebid.aliases?", impIndex, bidder)} } } + } else { + otherExtElements++ } } @@ -773,21 +829,30 @@ func (deps *endpointDeps) validateImpExt(imp *openrtb.Imp, aliases map[string]st } } - extJSON, err := json.Marshal(bidderExts) - if err != nil { - return []error{err} + if len(disabledBidders) > 0 || len(validationFailedBidders) > 0 { + extJSON, err := json.Marshal(bidderExts) + if err != nil { + return []error{err} + } + imp.Ext = extJSON } - imp.Ext = extJSON - // TODO #713 Fix this here - if len(bidderExts) < 1 { - errL = append(errL, fmt.Errorf("request.imp[%d].ext must contain at least one bidder with valid parameters", impIndex)) - return errL + if len(bidderExts)-otherExtElements == 0 { + errL = append(errL, fmt.Errorf("request.imp[%d].ext must contain at least one bidder", impIndex)) } return errL } +func isBidderToValidate(bidder string) bool { + // PrebidExtKey is a special case for the prebid config section and is not considered a bidder. + + // FirstPartyDataContextExtKey is a special case for the first party data context section + // and is not considered a bidder. + + return bidder != openrtb_ext.PrebidExtKey && bidder != openrtb_ext.FirstPartyDataContextExtKey +} + func (deps *endpointDeps) parseBidExt(ext json.RawMessage) (*openrtb_ext.ExtRequest, error) { if len(ext) < 1 { return nil, nil @@ -925,13 +990,27 @@ func validateRegs(regs *openrtb.Regs) error { return nil } +func sanitizeRequest(r *openrtb.BidRequest, ipValidator iputil.IPValidator) { + if r.Device != nil { + if ip, ver := iputil.ParseIP(r.Device.IP); ip == nil || ver != iputil.IPv4 || !ipValidator.IsValid(ip, ver) { + r.Device.IP = "" + } + + if ip, ver := iputil.ParseIP(r.Device.IPv6); ip == nil || ver != iputil.IPv6 || !ipValidator.IsValid(ip, ver) { + r.Device.IPv6 = "" + } + } +} + // setFieldsImplicitly uses _implicit_ information from the httpReq to set values on bidReq. // This function does not consume the request body, which was set explicitly, but infers certain // OpenRTB properties from the headers and other implicit info. // // This function _should not_ override any fields which were defined explicitly by the caller in the request. func (deps *endpointDeps) setFieldsImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest) { - setDeviceImplicitly(httpReq, bidReq) + sanitizeRequest(bidReq, deps.privateNetworkIPValidator) + + setDeviceImplicitly(httpReq, bidReq, deps.privateNetworkIPValidator) // Per the OpenRTB spec: A bid request must not contain both a Site and an App object. if bidReq.App == nil { @@ -943,9 +1022,11 @@ func (deps *endpointDeps) setFieldsImplicitly(httpReq *http.Request, bidReq *ope } // setDeviceImplicitly uses implicit info from httpReq to populate bidReq.Device -func setDeviceImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest) { - setIPImplicitly(httpReq, bidReq) // Fixes #230 +func setDeviceImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest, ipValidtor iputil.IPValidator) { + setIPImplicitly(httpReq, bidReq, ipValidtor) setUAImplicitly(httpReq, bidReq) + setDoNotTrackImplicitly(httpReq, bidReq) + } // setAuctionTypeImplicitly sets the auction type to 1 if it wasn't on the request, @@ -986,7 +1067,7 @@ func setSiteImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest) { func setImpsImplicitly(httpReq *http.Request, imps []openrtb.Imp) { secure := int8(1) for i := 0; i < len(imps); i++ { - if imps[i].Secure == nil && prebid.IsSecure(httpReq) { + if imps[i].Secure == nil && httputil.IsSecure(httpReq) { imps[i].Secure = &secure } } @@ -1143,13 +1224,21 @@ func getStoredRequestId(data []byte) (string, bool, error) { } // setIPImplicitly sets the IP address on bidReq, if it's not explicitly defined and we can figure it out. -func setIPImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest) { - if bidReq.Device == nil || bidReq.Device.IP == "" { - if ip := prebid.GetIP(httpReq); ip != "" { - if bidReq.Device == nil { - bidReq.Device = &openrtb.Device{} +func setIPImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest, ipValidator iputil.IPValidator) { + if bidReq.Device == nil || (bidReq.Device.IP == "" && bidReq.Device.IPv6 == "") { + if ip, ver := httputil.FindIP(httpReq, ipValidator); ip != nil { + switch ver { + case iputil.IPv4: + if bidReq.Device == nil { + bidReq.Device = &openrtb.Device{} + } + bidReq.Device.IP = ip.String() + case iputil.IPv6: + if bidReq.Device == nil { + bidReq.Device = &openrtb.Device{} + } + bidReq.Device.IPv6 = ip.String() } - bidReq.Device.IP = ip } } } @@ -1166,6 +1255,24 @@ func setUAImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest) { } } +func setDoNotTrackImplicitly(httpReq *http.Request, bidReq *openrtb.BidRequest) { + if bidReq.Device == nil || bidReq.Device.DNT == nil { + dnt := httpReq.Header.Get(dntKey) + if dnt == "0" || dnt == "1" { + if bidReq.Device == nil { + bidReq.Device = &openrtb.Device{} + } + + switch dnt { + case "0": + bidReq.Device.DNT = &dntDisabled + case "1": + bidReq.Device.DNT = &dntEnabled + } + } + } +} + // parseUserID gets this user's ID for the host machine, if it exists. func parseUserID(cfg *config.Configuration, httpReq *http.Request) (string, bool) { if hostCookie, err := httpReq.Cookie(cfg.HostCookie.CookieName); hostCookie != nil && err == nil { @@ -1213,8 +1320,8 @@ func writeError(errs []error, w http.ResponseWriter, labels *pbsmetrics.Labels) return rc } -// Returns the effective publisher ID -func effectivePubID(pub *openrtb.Publisher) string { +// Returns the account ID for the request +func getAccountID(pub *openrtb.Publisher) string { if pub != nil { if pub.Ext != nil { var pubExt openrtb_ext.ExtPublisher @@ -1229,15 +1336,3 @@ func effectivePubID(pub *openrtb.Publisher) string { } return pbsmetrics.PublisherUnknown } - -func validateAccount(cfg *config.Configuration, pubID string) error { - var err error = nil - if cfg.AccountRequired && pubID == pbsmetrics.PublisherUnknown { - // If specified in the configuration, discard requests that don't come with an account ID. - err = error(&errortypes.AcctRequired{Message: fmt.Sprintf("Prebid-server has been configured to discard requests that don't come with an Account ID. Please reach out to the prebid server host.")}) - } else if _, found := cfg.BlacklistedAcctMap[pubID]; found { - // Blacklist account now that we have resolved the value - err = error(&errortypes.BlacklistedAcct{Message: fmt.Sprintf("Prebid-server has blacklisted Account ID: %s, please reach out to the prebid server host.", pubID)}) - } - return err -} diff --git a/endpoints/openrtb2/auction_benchmark_test.go b/endpoints/openrtb2/auction_benchmark_test.go index 2ac82b5c52f..ad50a02805e 100644 --- a/endpoints/openrtb2/auction_benchmark_test.go +++ b/endpoints/openrtb2/auction_benchmark_test.go @@ -5,6 +5,7 @@ import ( "net/http/httptest" "strings" "testing" + "time" "github.com/PubMatic-OpenWrap/prebid-server/adapters" "github.com/PubMatic-OpenWrap/prebid-server/currencies" @@ -77,7 +78,8 @@ func BenchmarkOpenrtbEndpoint(b *testing.B) { theMetrics, infos, gdpr.AlwaysAllow{}, - currencies.NewRateConverterDefault(), + currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)), + empty_fetcher.EmptyFetcher{}, ), paramValidator, empty_fetcher.EmptyFetcher{}, diff --git a/endpoints/openrtb2/auction_test.go b/endpoints/openrtb2/auction_test.go index 07d477a3730..798e59e5bf9 100644 --- a/endpoints/openrtb2/auction_test.go +++ b/endpoints/openrtb2/auction_test.go @@ -8,9 +8,11 @@ import ( "fmt" "io" "io/ioutil" + "net" "net/http" "net/http/httptest" "os" + "path/filepath" "strings" "testing" "time" @@ -27,6 +29,7 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/empty_fetcher" + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" "github.com/buger/jsonparser" jsonpatch "github.com/evanphx/json-patch" "github.com/stretchr/testify/assert" @@ -34,18 +37,325 @@ import ( const maxSize = 1024 * 256 -// Struct of data for the general purpose auction tester -type getResponseFromDirectory struct { - dir string - file string - payloadGetter func(*testing.T, []byte) []byte - messageGetter func(*testing.T, []byte) []byte - expectedCode int - aliased bool - disabledBidders []string - adaptersConfig map[string]config.Adapter - accountReq bool - description string +type testCase struct { + BidRequest json.RawMessage `json:"mockBidRequest"` + Config *testConfigValues `json:"config"` + ExpectedReturnCode int `json:"expectedReturnCode,omitempty"` + ExpectedErrorMessage string `json:"expectedErrorMessage"` + ExpectedBidResponse json.RawMessage `json:"expectedBidResponse"` +} + +type testConfigValues struct { + AccountRequired bool `json:"accountRequired"` + AliasJSON string `json:"aliases"` + BlacklistedAccounts []string `json:"blacklistedAccts"` + BlacklistedApps []string `json:"blacklistedApps"` + AdapterList []string `json:"disabledAdapters"` +} + +func TestJsonSampleRequests(t *testing.T) { + testSuites := []struct { + description string + sampleRequestsSubDir string + }{ + { + "Assert 200s on all bidRequests from exemplary folder", + "valid-whole/exemplary", + }, + { + "Asserts we return 200s on well-formed Native requests.", + "valid-native", + }, + { + "Asserts we return 400s on requests that are not supposed to pass validation", + "invalid-whole", + }, + { + "Asserts we return 400s on requests with Native requests that don't pass validation", + "invalid-native", + }, + { + "Makes sure we handle (default) aliased bidders properly", + "aliased", + }, + { + "Asserts we return 503s on requests with blacklisted accounts and apps.", + "blacklisted", + }, + { + "Assert that requests that come with no user id nor app id return error if the `AccountRequired` field in the `config.Configuration` structure is set to true", + "account-required/no-account", + }, + { + "Assert requests that come with a valid user id or app id when account is required", + "account-required/with-account", + }, + { + "Tests diagnostic messages for invalid stored requests", + "invalid-stored", + }, + { + "Make sure requests with disabled bidders will fail", + "disabled/bad", + }, + { + "There are both disabled and non-disabled bidders, we expect a 200", + "disabled/good", + }, + { + "Requests with first party data context info found in imp[i].ext.prebid.bidder,context", + "first-party-data", + }, + } + for _, test := range testSuites { + testCaseFiles, err := getTestFiles(filepath.Join("sample-requests", test.sampleRequestsSubDir)) + if assert.NoError(t, err, "Test case %s. Error reading files from directory %s \n", test.description, test.sampleRequestsSubDir) { + for _, file := range testCaseFiles { + data, err := ioutil.ReadFile(file) + if assert.NoError(t, err, "Test case %s. Error reading file %s \n", test.description, file) { + runTestCase(t, data, file) + } + } + } + } +} + +func getTestFiles(dir string) ([]string, error) { + var filesToAssert []string + + fileList, err := ioutil.ReadDir(dir) + if err != nil { + return nil, err + } + + // Append the path of every file found in `dir` to the `filesToAssert` array + for _, fileInfo := range fileList { + filesToAssert = append(filesToAssert, filepath.Join(dir, fileInfo.Name())) + } + + return filesToAssert, nil +} + +func runTestCase(t *testing.T, fileData []byte, testFile string) { + t.Helper() + + // Retrieve values from JSON file + test := parseTestFile(t, fileData, testFile) + + // Run test + actualCode, actualJsonBidResponse := doRequest(t, test) + + // Assertions + assert.Equal(t, test.ExpectedReturnCode, actualCode, "Test failed. Filename: %s \n", testFile) + + // Either assert bid response or expected error + if len(test.ExpectedErrorMessage) > 0 { + assert.True(t, strings.HasPrefix(actualJsonBidResponse, test.ExpectedErrorMessage), "Actual: %s \nExpected: %s. Filename: %s \n", actualJsonBidResponse, test.ExpectedErrorMessage, testFile) + } + + if len(test.ExpectedBidResponse) > 0 { + var expectedBidResponse openrtb.BidResponse + var actualBidResponse openrtb.BidResponse + var err error + + err = json.Unmarshal(test.ExpectedBidResponse, &expectedBidResponse) + if assert.NoError(t, err, "Could not unmarshal expected bidResponse taken from test file.\n Test file: %s\n Error:%s\n", testFile, err) { + err = json.Unmarshal([]byte(actualJsonBidResponse), &actualBidResponse) + if assert.NoError(t, err, "Could not unmarshal actual bidResponse from auction.\n Test file: %s\n Error:%s\n", testFile, err) { + assertBidResponseEqual(t, testFile, expectedBidResponse, actualBidResponse) + } + } + } +} + +func parseTestFile(t *testing.T, fileData []byte, testFile string) testCase { + t.Helper() + + parsedTestData := testCase{} + var err, errEm error + + // Get testCase values + parsedTestData.BidRequest, _, _, err = jsonparser.Get(fileData, "mockBidRequest") + assert.NoError(t, err, "Error jsonparsing root.mockBidRequest from file %s. Desc: %v.", testFile, err) + + // Get testCaseConfig values + parsedTestData.Config = &testConfigValues{} + var jsonTestConfig json.RawMessage + + jsonTestConfig, _, _, err = jsonparser.Get(fileData, "config") + if err == nil { + err = json.Unmarshal(jsonTestConfig, parsedTestData.Config) + assert.NoError(t, err, "Error unmarshaling root.config from file %s. Desc: %v.", testFile, err) + } + + // Get the return code we expect PBS to throw back given test's bidRequest and config + parsedReturnCode, err := jsonparser.GetInt(fileData, "expectedReturnCode") + assert.NoError(t, err, "Error jsonparsing root.code from file %s. Desc: %v.", testFile, err) + + // Get both bid response and error message, if any + parsedTestData.ExpectedBidResponse, _, _, err = jsonparser.Get(fileData, "expectedBidResponse") + parsedTestData.ExpectedErrorMessage, errEm = jsonparser.GetString(fileData, "expectedErrorMessage") + + assert.Falsef(t, (err == nil && errEm == nil), "Test case file can't have both a valid expectedBidResponse and a valid expectedErrorMessage, fields are mutually exclusive") + assert.Falsef(t, (err != nil && errEm != nil), "Test case file should come with either a valid expectedBidResponse or a valid expectedErrorMessage, not both.") + + parsedTestData.ExpectedReturnCode = int(parsedReturnCode) + + return parsedTestData +} + +func (tc *testConfigValues) getBlacklistedAppMap() map[string]bool { + var blacklistedAppMap map[string]bool + + if len(tc.BlacklistedApps) > 0 { + blacklistedAppMap = make(map[string]bool, len(tc.BlacklistedApps)) + for _, app := range tc.BlacklistedApps { + blacklistedAppMap[app] = true + } + } + return blacklistedAppMap +} + +func (tc *testConfigValues) getBlackListedAccountMap() map[string]bool { + var blacklistedAccountMap map[string]bool + + if len(tc.BlacklistedAccounts) > 0 { + blacklistedAccountMap = make(map[string]bool, len(tc.BlacklistedAccounts)) + for _, account := range tc.BlacklistedAccounts { + blacklistedAccountMap[account] = true + } + } + return blacklistedAccountMap +} + +func (tc *testConfigValues) getAdaptersConfigMap() map[string]config.Adapter { + var adaptersConfig map[string]config.Adapter + + if len(tc.AdapterList) > 0 { + adaptersConfig = make(map[string]config.Adapter, len(tc.AdapterList)) + for _, adapterName := range tc.AdapterList { + adaptersConfig[adapterName] = config.Adapter{Disabled: true} + } + } + return adaptersConfig +} + +// Once unmarshalled, bidResponse objects can't simply be compared with an `assert.Equalf()` call +// because tests fail if the elements inside the `bidResponse.SeatBid` and `bidResponse.SeatBid.Bid` +// arrays, if any, are not listed in the exact same order in the actual version and in the expected version. +func assertBidResponseEqual(t *testing.T, testFile string, expectedBidResponse openrtb.BidResponse, actualBidResponse openrtb.BidResponse) { + + //Assert non-array BidResponse fields + assert.Equalf(t, expectedBidResponse.ID, actualBidResponse.ID, "BidResponse.ID doesn't match expected. Test: %s\n", testFile) + assert.Equalf(t, expectedBidResponse.BidID, actualBidResponse.BidID, "BidResponse.BidID doesn't match expected. Test: %s\n", testFile) + assert.Equalf(t, expectedBidResponse.NBR, actualBidResponse.NBR, "BidResponse.NBR doesn't match expected. Test: %s\n", testFile) + + //Assert []SeatBid and their Bid elements independently of their order + assert.Len(t, actualBidResponse.SeatBid, len(expectedBidResponse.SeatBid), "BidResponse.SeatBid array doesn't match expected. Test: %s\n", testFile) + + //Given that bidResponses have the same length, compare them in an order-independent way using maps + var actualSeatBidsMap map[string]openrtb.SeatBid = make(map[string]openrtb.SeatBid, 0) + for _, seatBid := range actualBidResponse.SeatBid { + actualSeatBidsMap[seatBid.Seat] = seatBid + } + + var expectedSeatBidsMap map[string]openrtb.SeatBid = make(map[string]openrtb.SeatBid, 0) + for _, seatBid := range expectedBidResponse.SeatBid { + expectedSeatBidsMap[seatBid.Seat] = seatBid + } + + for k, expectedSeatBid := range expectedSeatBidsMap { + //Assert non-array SeatBid fields + assert.Equalf(t, expectedSeatBid.Seat, actualSeatBidsMap[k].Seat, "actualSeatBidsMap[%s].Seat doesn't match expected. Test: %s\n", k, testFile) + assert.Equalf(t, expectedSeatBid.Group, actualSeatBidsMap[k].Group, "actualSeatBidsMap[%s].Group doesn't match expected. Test: %s\n", k, testFile) + assert.Equalf(t, expectedSeatBid.Ext, actualSeatBidsMap[k].Ext, "actualSeatBidsMap[%s].Ext doesn't match expected. Test: %s\n", k, testFile) + assert.Len(t, actualSeatBidsMap[k].Bid, len(expectedSeatBid.Bid), "BidResponse.SeatBid[].Bid array doesn't match expected. Test: %s\n", testFile) + + //Assert Bid arrays + assert.ElementsMatch(t, expectedSeatBid.Bid, actualSeatBidsMap[k].Bid, "BidResponse.SeatBid array doesn't match expected. Test: %s\n", testFile) + } +} + +func TestBidRequestAssert(t *testing.T) { + appnexusB1 := openrtb.Bid{ID: "appnexus-bid-1", Price: 5.00} + appnexusB2 := openrtb.Bid{ID: "appnexus-bid-2", Price: 7.00} + rubiconB1 := openrtb.Bid{ID: "rubicon-bid-1", Price: 1.50} + rubiconB2 := openrtb.Bid{ID: "rubicon-bid-2", Price: 4.00} + + sampleSeatBids := []openrtb.SeatBid{ + { + Seat: "appnexus-bids", + Bid: []openrtb.Bid{appnexusB1, appnexusB2}, + }, + { + Seat: "rubicon-bids", + Bid: []openrtb.Bid{rubiconB1, rubiconB2}, + }, + } + + testSuites := []struct { + description string + expectedBidResponse openrtb.BidResponse + actualBidResponse openrtb.BidResponse + }{ + { + "identical SeatBids, exact same SeatBid and Bid arrays order", + openrtb.BidResponse{ID: "anId", BidID: "bidId", SeatBid: sampleSeatBids}, + openrtb.BidResponse{ID: "anId", BidID: "bidId", SeatBid: sampleSeatBids}, + }, + { + "identical SeatBids but Seatbid array elements come in different order", + openrtb.BidResponse{ID: "anId", BidID: "bidId", SeatBid: sampleSeatBids}, + openrtb.BidResponse{ID: "anId", BidID: "bidId", + SeatBid: []openrtb.SeatBid{ + { + Seat: "rubicon-bids", + Bid: []openrtb.Bid{rubiconB1, rubiconB2}, + }, + { + Seat: "appnexus-bids", + Bid: []openrtb.Bid{appnexusB1, appnexusB2}, + }, + }, + }, + }, + { + "SeatBids seem to be identical except for the different order of Bid array elements in one of them", + openrtb.BidResponse{ID: "anId", BidID: "bidId", SeatBid: sampleSeatBids}, + openrtb.BidResponse{ID: "anId", BidID: "bidId", + SeatBid: []openrtb.SeatBid{ + { + Seat: "appnexus-bids", + Bid: []openrtb.Bid{appnexusB2, appnexusB1}, + }, + { + Seat: "rubicon-bids", + Bid: []openrtb.Bid{rubiconB1, rubiconB2}, + }, + }, + }, + }, + { + "Both SeatBid elements and bid elements come in different order", + openrtb.BidResponse{ID: "anId", BidID: "bidId", SeatBid: sampleSeatBids}, + openrtb.BidResponse{ID: "anId", BidID: "bidId", + SeatBid: []openrtb.SeatBid{ + { + Seat: "rubicon-bids", + Bid: []openrtb.Bid{rubiconB2, rubiconB1}, + }, + { + Seat: "appnexus-bids", + Bid: []openrtb.Bid{appnexusB2, appnexusB1}, + }, + }, + }, + }, + } + + for _, test := range testSuites { + assertBidResponseEqual(t, test.description, test.expectedBidResponse, test.actualBidResponse) + } } // TestExplicitUserId makes sure that the cookie's ID doesn't override an explicit value sent in the request. @@ -61,7 +371,7 @@ func TestExplicitUserId(t *testing.T) { ex := &mockExchange{} request := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(`{ - "id": "some-request-id", +"id": "some-request-id", "site": { "page": "test.somepage.com" }, @@ -118,198 +428,38 @@ func TestExplicitUserId(t *testing.T) { } } -// TestGoodRequests makes sure we return 200s on good requests. -func TestGoodRequests(t *testing.T) { - exemplary := &getResponseFromDirectory{ - dir: "sample-requests/valid-whole/exemplary", - payloadGetter: getRequestPayload, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - aliased: true, - } - supplementary := &getResponseFromDirectory{ - dir: "sample-requests/valid-whole/supplementary", - payloadGetter: noop, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - aliased: true, - } - exemplary.assert(t) - supplementary.assert(t) -} - -// TestGoodNativeRequests makes sure we return 200s on well-formed Native requests. -func TestGoodNativeRequests(t *testing.T) { - tests := &getResponseFromDirectory{ - dir: "sample-requests/valid-native", - payloadGetter: buildNativeRequest, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - aliased: true, - } - tests.assert(t) -} - -// TestBadRequests makes sure we return 400s on bad requests. -func TestBadRequests(t *testing.T) { - // Need to turn off aliases for bad requests as applying the alias can fail on a bad request before the expected error is reached. - tests := &getResponseFromDirectory{ - dir: "sample-requests/invalid-whole", - payloadGetter: getRequestPayload, - messageGetter: getMessage, - expectedCode: http.StatusBadRequest, - aliased: false, - } - tests.assert(t) -} - -// TestBadRequests makes sure we return 400s on requests with bad Native requests. -func TestBadNativeRequests(t *testing.T) { - tests := &getResponseFromDirectory{ - dir: "sample-requests/invalid-native", - payloadGetter: buildNativeRequest, - messageGetter: nilReturner, - expectedCode: http.StatusBadRequest, - aliased: false, - } - tests.assert(t) -} - -// TestAliasedRequests makes sure we handle (default) aliased bidders properly -func TestAliasedRequests(t *testing.T) { - tests := &getResponseFromDirectory{ - dir: "sample-requests/aliased", - payloadGetter: noop, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - aliased: true, - } - tests.assert(t) -} - -// TestDisabledBidders makes sure we don't break when encountering a disabled bidder -func TestDisabledBidders(t *testing.T) { - badTests := &getResponseFromDirectory{ - dir: "sample-requests/disabled/bad", - payloadGetter: getRequestPayload, - messageGetter: getMessage, - expectedCode: http.StatusBadRequest, - aliased: false, - disabledBidders: []string{"appnexus", "rubicon"}, - adaptersConfig: map[string]config.Adapter{ - "appnexus": {Disabled: true}, - "rubicon": {Disabled: true}, - }, - } - goodTests := &getResponseFromDirectory{ - dir: "sample-requests/disabled/good", - payloadGetter: noop, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - aliased: false, - disabledBidders: []string{"appnexus", "rubicon"}, - adaptersConfig: map[string]config.Adapter{ - "appnexus": {Disabled: true}, - "rubicon": {Disabled: true}, - }, - } - badTests.assert(t) - goodTests.assert(t) -} - -// TestBlacklistRequests makes sure we return 400s on blacklisted requests. -func TestBlacklistRequests(t *testing.T) { - // Need to turn off aliases for bad requests as applying the alias can fail on a bad request before the expected error is reached. - tests := &getResponseFromDirectory{ - dir: "sample-requests/blacklisted", - payloadGetter: getRequestPayload, - messageGetter: getMessage, - expectedCode: http.StatusServiceUnavailable, - aliased: false, - } - tests.assert(t) -} - -// TestRejectAccountRequired asserts we return a 400 code on a request that comes with no user id nor app id -// if the `AccountRequired` field in the `config.Configuration` structure is set to true -func TestRejectAccountRequired(t *testing.T) { - tests := []*getResponseFromDirectory{ - { - // Account not required and not provided in prebid request - dir: "sample-requests/account-required", - file: "no-acct.json", - payloadGetter: getRequestPayload, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - accountReq: false, - }, - { - // Account was required but not provided in prebid request - dir: "sample-requests/account-required", - file: "no-acct.json", - payloadGetter: getRequestPayload, - messageGetter: getMessage, - expectedCode: http.StatusBadRequest, - accountReq: true, - }, - { - // Account is required, was provided and is not in the blacklisted accounts map - dir: "sample-requests/account-required", - file: "with-acct.json", - payloadGetter: getRequestPayload, - messageGetter: nilReturner, - expectedCode: http.StatusOK, - aliased: true, - accountReq: true, - }, - { - // Account is required, was provided in request and is found in the blacklisted accounts map - dir: "sample-requests/blacklisted", - file: "blacklisted-acct.json", - payloadGetter: getRequestPayload, - messageGetter: getMessage, - expectedCode: http.StatusServiceUnavailable, - accountReq: true, +func doRequest(t *testing.T, test testCase) (int, string) { + disabledBidders := map[string]string{} + bidderMap := exchange.DisableBidders(getBidderInfos(test.Config.getAdaptersConfigMap(), openrtb_ext.BidderList()), disabledBidders) + + // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. + // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + + endpoint, _ := NewEndpoint( + &mockBidExchange{}, + newParamsValidator(t), + &mockStoredReqFetcher{}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{ + MaxRequestSize: maxSize, + BlacklistedApps: test.Config.BlacklistedApps, + BlacklistedAppMap: test.Config.getBlacklistedAppMap(), + BlacklistedAccts: test.Config.BlacklistedAccounts, + BlacklistedAcctMap: test.Config.getBlackListedAccountMap(), + AccountRequired: test.Config.AccountRequired, }, - } - for _, test := range tests { - test.assert(t) - } -} + metrics, + analyticsConf.NewPBSAnalytics(&config.Analytics{}), + disabledBidders, + []byte(test.Config.AliasJSON), + bidderMap, + ) -// assertResponseFromDirectory makes sure that the payload from each file in dir gets the expected response status code -// from the /openrtb2/auction endpoint. -func (gr *getResponseFromDirectory) assert(t *testing.T) { - //t *testing.T, dir string, payloadGetter func(*testing.T, []byte) []byte, messageGetter func(*testing.T, []byte) []byte, expectedCode int, aliased bool) { - t.Helper() - var filesToAssert []string - if gr.file == "" { - // Append every file found in `gr.dir` to the `filesToAssert` array and test them all - for _, fileInfo := range fetchFiles(t, gr.dir) { - filesToAssert = append(filesToAssert, gr.dir+"/"+fileInfo.Name()) - } - } else { - // Just test the single `gr.file`, and not the entirety of files that may be found in `gr.dir` - filesToAssert = append(filesToAssert, gr.dir+"/"+gr.file) - } - - var fileData []byte - // Test the one or more test files appended to `filesToAssert` - for _, testFile := range filesToAssert { - fileData = readFile(t, testFile) - code, msg := gr.doRequest(t, gr.payloadGetter(t, fileData)) - fmt.Printf("Processing %s\n", testFile) - assertResponseCode(t, testFile, code, gr.expectedCode, msg) - - expectMsg := gr.messageGetter(t, fileData) - if gr.description != "" { - if len(expectMsg) > 0 { - assert.Equal(t, string(expectMsg), msg, "Test failed. %s. Filename: \n", gr.description, testFile) - } else { - assert.Equal(t, string(expectMsg), msg, "file %s had bad response body", testFile) - } - } - } + request := httptest.NewRequest("POST", "/openrtb2/auction", bytes.NewReader(test.BidRequest)) + recorder := httptest.NewRecorder() + endpoint(recorder, request, nil) //Request comes from the unmarshalled mockBidRequest + return recorder.Code, recorder.Body.String() } // fetchFiles returns a list of the files from dir, or fails the test if an error occurs. @@ -330,39 +480,6 @@ func readFile(t *testing.T, filename string) []byte { return data } -// doRequest populates the app with mock dependencies and sends requestData to the /openrtb2/auction endpoint. -func (gr *getResponseFromDirectory) doRequest(t *testing.T, requestData []byte) (int, string) { - aliasJSON := []byte{} - if gr.aliased { - aliasJSON = []byte(`{"ext":{"prebid":{"aliases": {"test1": "appnexus", "test2": "rubicon", "test3": "openx"}}}}`) - } - disabledBidders := map[string]string{ - "indexExchange": "Bidder \"indexExchange\" has been deprecated and is no longer available. Please use bidder \"ix\" and note that the bidder params have changed.", - } - bidderMap := exchange.DisableBidders(getBidderInfos(gr.adaptersConfig, openrtb_ext.BidderList()), disabledBidders) - - // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. - // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - endpoint, _ := NewEndpoint( - &nobidExchange{}, - newParamsValidator(t), - &mockStoredReqFetcher{}, - empty_fetcher.EmptyFetcher{}, - &config.Configuration{MaxRequestSize: maxSize, BlacklistedApps: []string{"spam_app"}, BlacklistedAppMap: map[string]bool{"spam_app": true}, BlacklistedAccts: []string{"bad_acct"}, BlacklistedAcctMap: map[string]bool{"bad_acct": true}, AccountRequired: gr.accountReq}, - theMetrics, - analyticsConf.NewPBSAnalytics(&config.Analytics{}), - disabledBidders, - aliasJSON, - bidderMap, - ) - - request := httptest.NewRequest("POST", "/openrtb2/auction", bytes.NewReader(requestData)) - recorder := httptest.NewRecorder() - endpoint(recorder, request, nil) - return recorder.Code, recorder.Body.String() -} - // TestBadAliasRequests() reuses two requests that would fail anyway. Here, we // take advantage of our knowledge that processStoredRequests() in auction.go // processes aliases before it processes stored imps. Changing that order @@ -376,7 +493,9 @@ func TestBadAliasRequests(t *testing.T) { func doBadAliasRequest(t *testing.T, filename string, expectMsg string) { t.Helper() fileData := readFile(t, filename) - requestData := getRequestPayload(t, fileData) + testBidRequest, _, _, err := jsonparser.Get(fileData, "mockBidRequest") + assert.NoError(t, err, "Error jsonparsing root.mockBidRequest from file %s. Desc: %v.", filename, err) + // aliasJSON lacks a comma after the "appnexus" entry so is bad JSON aliasJSON := []byte(`{"ext":{"prebid":{"aliases": {"test1": "appnexus" "test2": "rubicon", "test3": "openx"}}}}`) disabledBidders := map[string]string{ @@ -387,10 +506,10 @@ func doBadAliasRequest(t *testing.T, filename string, expectMsg string) { // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - endpoint, _ := NewEndpoint(&nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, theMetrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), disabledBidders, aliasJSON, bidderMap) + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + endpoint, _ := NewEndpoint(&nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, metrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), disabledBidders, aliasJSON, bidderMap) - request := httptest.NewRequest("POST", "/openrtb2/auction", bytes.NewReader(requestData)) + request := httptest.NewRequest("POST", "/openrtb2/auction", bytes.NewReader(testBidRequest)) recorder := httptest.NewRecorder() endpoint(recorder, request, nil) @@ -414,29 +533,6 @@ func assertResponseCode(t *testing.T, filename string, actual int, expected int, } } -// buildNativeRequest JSON-encodes the nativeData as a string, and puts it into request.imp[0].native.request -// of a request which is valid otherwise. -func buildNativeRequest(t *testing.T, nativeData []byte) []byte { - serialized, err := json.Marshal(string(nativeData)) - if err != nil { - t.Fatalf("Failed to string-escape JSON data: %v", err) - } - - buf := bytes.NewBuffer(nil) - buf.WriteString(`{"id":"req-id","site":{"page":"some.page.com"},"tmax":500,"imp":[{"id":"some-imp","native":{"request":`) - buf.Write(serialized) - buf.WriteString(`},"ext":{"appnexus":{"placementId":12883451}}}]}`) - return buf.Bytes() -} - -func noop(t *testing.T, data []byte) []byte { - return data -} - -func nilReturner(t *testing.T, data []byte) []byte { - return nil -} - func getRequestPayload(t *testing.T, example []byte) []byte { t.Helper() if value, _, _, err := jsonparser.Get(example, "requestPayload"); err != nil { @@ -451,8 +547,8 @@ func getRequestPayload(t *testing.T, example []byte) []byte { func TestNilExchange(t *testing.T) { // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - _, err := NewEndpoint(nil, newParamsValidator(t), empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, theMetrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + _, err := NewEndpoint(nil, newParamsValidator(t), empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, metrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) if err == nil { t.Errorf("NewEndpoint should return an error when given a nil Exchange.") } @@ -462,8 +558,8 @@ func TestNilExchange(t *testing.T) { func TestNilValidator(t *testing.T) { // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - _, err := NewEndpoint(&nobidExchange{}, nil, empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, theMetrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + _, err := NewEndpoint(&nobidExchange{}, nil, empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, metrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) if err == nil { t.Errorf("NewEndpoint should return an error when given a nil BidderParamValidator.") } @@ -473,8 +569,8 @@ func TestNilValidator(t *testing.T) { func TestExchangeError(t *testing.T) { // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - endpoint, _ := NewEndpoint(&brokenExchange{}, newParamsValidator(t), empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, theMetrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + endpoint, _ := NewEndpoint(&brokenExchange{}, newParamsValidator(t), empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, metrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) request := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, "site.json"))) recorder := httptest.NewRecorder() endpoint(recorder, request, nil) @@ -526,29 +622,265 @@ func TestAuctionTypeDefault(t *testing.T) { } } -// TestImplicitIPs prevents #230 -func TestImplicitIPs(t *testing.T) { - ex := &nobidExchange{} - // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. - // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - endpoint, _ := NewEndpoint(ex, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, theMetrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) +func TestImplicitIPsEndToEnd(t *testing.T) { + testCases := []struct { + description string + reqJSONFile string + xForwardedForHeader string + privateNetworksIPv4 []net.IPNet + privateNetworksIPv6 []net.IPNet + expectedDeviceIPv4 string + expectedDeviceIPv6 string + }{ + { + description: "IPv4", + reqJSONFile: "site.json", + xForwardedForHeader: "1.1.1.1", + expectedDeviceIPv4: "1.1.1.1", + }, + { + description: "IPv6", + reqJSONFile: "site.json", + xForwardedForHeader: "1111::", + expectedDeviceIPv6: "1111::", + }, + { + description: "IPv4 - Defined In Request", + reqJSONFile: "site-has-ipv4.json", + xForwardedForHeader: "1.1.1.1", + expectedDeviceIPv4: "8.8.8.8", // Hardcoded value in test file. + }, + { + description: "IPv6 - Defined In Request", + reqJSONFile: "site-has-ipv6.json", + xForwardedForHeader: "1111::", + expectedDeviceIPv6: "8888::", // Hardcoded value in test file. + }, + { + description: "IPv4 - Defined In Request - Private Network", + reqJSONFile: "site-has-ipv4.json", + xForwardedForHeader: "1.1.1.1", + privateNetworksIPv4: []net.IPNet{{IP: net.IP{8, 8, 8, 0}, Mask: net.CIDRMask(24, 32)}}, // Hardcoded value in test file. + expectedDeviceIPv4: "1.1.1.1", + }, + { + description: "IPv6 - Defined In Request - Private Network", + reqJSONFile: "site-has-ipv6.json", + xForwardedForHeader: "1111::", + privateNetworksIPv6: []net.IPNet{{IP: net.ParseIP("8800::"), Mask: net.CIDRMask(8, 128)}}, // Hardcoded value in test file. + expectedDeviceIPv6: "1111::", + }, + } - httpReq := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, "site.json"))) - httpReq.Header.Set("X-Forwarded-For", "123.456.78.90") - recorder := httptest.NewRecorder() + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + for _, test := range testCases { + exchange := &nobidExchange{} + cfg := &config.Configuration{ + MaxRequestSize: maxSize, + RequestValidation: config.RequestValidation{ + IPv4PrivateNetworksParsed: test.privateNetworksIPv4, + IPv6PrivateNetworksParsed: test.privateNetworksIPv6, + }, + } + endpoint, _ := NewEndpoint(exchange, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, cfg, metrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) - endpoint(recorder, httpReq, nil) + httpReq := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, test.reqJSONFile))) + httpReq.Header.Set("X-Forwarded-For", test.xForwardedForHeader) - if ex.gotRequest == nil { - t.Fatalf("The request never made it into the Exchange.") + endpoint(httptest.NewRecorder(), httpReq, nil) + + result := exchange.gotRequest + if !assert.NotEmpty(t, result, test.description+"Request received by the exchange.") { + t.FailNow() + } + assert.Equal(t, test.expectedDeviceIPv4, result.Device.IP, test.description+":ipv4") + assert.Equal(t, test.expectedDeviceIPv6, result.Device.IPv6, test.description+":ipv6") } +} - if ex.gotRequest.Device.IP != "123.456.78.90" { - t.Errorf("Bad device IP. Expected 123.456.78.90, got %s", ex.gotRequest.Device.IP) +func TestImplicitDNT(t *testing.T) { + var ( + disabled int8 = 0 + enabled int8 = 1 + ) + testCases := []struct { + description string + dntHeader string + request openrtb.BidRequest + expectedRequest openrtb.BidRequest + }{ + { + description: "Device Missing - Not Set In Header", + dntHeader: "", + request: openrtb.BidRequest{}, + expectedRequest: openrtb.BidRequest{}, + }, + { + description: "Device Missing - Set To 0 In Header", + dntHeader: "0", + request: openrtb.BidRequest{}, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &disabled, + }, + }, + }, + { + description: "Device Missing - Set To 1 In Header", + dntHeader: "1", + request: openrtb.BidRequest{}, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + }, + { + description: "Not Set In Request - Not Set In Header", + dntHeader: "", + request: openrtb.BidRequest{ + Device: &openrtb.Device{}, + }, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{}, + }, + }, + { + description: "Not Set In Request - Set To 0 In Header", + dntHeader: "0", + request: openrtb.BidRequest{ + Device: &openrtb.Device{}, + }, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &disabled, + }, + }, + }, + { + description: "Not Set In Request - Set To 1 In Header", + dntHeader: "1", + request: openrtb.BidRequest{ + Device: &openrtb.Device{}, + }, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + }, + { + description: "Set In Request - Not Set In Header", + dntHeader: "", + request: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + }, + { + description: "Set In Request - Set To 0 In Header", + dntHeader: "0", + request: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + }, + { + description: "Set In Request - Set To 1 In Header", + dntHeader: "1", + request: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + expectedRequest: openrtb.BidRequest{ + Device: &openrtb.Device{ + DNT: &enabled, + }, + }, + }, + } + + for _, test := range testCases { + httpReq := httptest.NewRequest("POST", "/openrtb2/auction", nil) + httpReq.Header.Set("DNT", test.dntHeader) + setDoNotTrackImplicitly(httpReq, &test.request) + assert.Equal(t, test.expectedRequest, test.request) } } +func TestImplicitDNTEndToEnd(t *testing.T) { + var ( + disabled int8 = 0 + enabled int8 = 1 + ) + testCases := []struct { + description string + reqJSONFile string + dntHeader string + expectedDNT *int8 + }{ + { + description: "Not Set In Request - Not Set In Header", + reqJSONFile: "site.json", + dntHeader: "", + expectedDNT: nil, + }, + { + description: "Not Set In Request - Set To 0 In Header", + reqJSONFile: "site.json", + dntHeader: "0", + expectedDNT: &disabled, + }, + { + description: "Not Set In Request - Set To 1 In Header", + reqJSONFile: "site.json", + dntHeader: "1", + expectedDNT: &enabled, + }, + { + description: "Set In Request - Not Set In Header", + reqJSONFile: "site-has-dnt.json", + dntHeader: "", + expectedDNT: &enabled, // Hardcoded value in test file. + }, + { + description: "Set In Request - Not Overwritten By Header", + reqJSONFile: "site-has-dnt.json", + dntHeader: "0", + expectedDNT: &enabled, // Hardcoded value in test file. + }, + } + + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + for _, test := range testCases { + exchange := &nobidExchange{} + endpoint, _ := NewEndpoint(exchange, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, metrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, []byte{}, openrtb_ext.BidderMap) + + httpReq := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, test.reqJSONFile))) + httpReq.Header.Set("DNT", test.dntHeader) + + endpoint(httptest.NewRecorder(), httpReq, nil) + + result := exchange.gotRequest + if !assert.NotEmpty(t, result, test.description+"Request received by the exchange.") { + t.FailNow() + } + assert.Equal(t, test.expectedDNT, result.Device.DNT, test.description+":dnt") + } +} func TestImplicitSecure(t *testing.T) { httpReq := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, "site.json"))) httpReq.Header.Set(http.CanonicalHeaderKey("X-Forwarded-Proto"), "https") @@ -584,28 +916,31 @@ func TestRefererParsing(t *testing.T) { } } -// TestBadStoredRequests tests diagnostic messages for invalid stored requests -func TestBadStoredRequests(t *testing.T) { - // Need to turn off aliases for bad requests as applying the alias can fail on a bad request before the expected error is reached. - tests := &getResponseFromDirectory{ - dir: "sample-requests/invalid-stored", - payloadGetter: getRequestPayload, - messageGetter: getMessage, - expectedCode: http.StatusBadRequest, - aliased: false, - } - tests.assert(t) -} - // Test the stored request functionality func TestStoredRequests(t *testing.T) { // NewMetrics() will create a new go_metrics MetricsEngine, bypassing the need for a crafted configuration set to support it. // As a side effect this gives us some coverage of the go_metrics piece of the metrics engine. - theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - edep := &endpointDeps{&nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, &config.Configuration{MaxRequestSize: maxSize}, theMetrics, analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, false, []byte{}, openrtb_ext.BidderMap, nil, nil} + metrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + deps := &endpointDeps{ + &nobidExchange{}, + newParamsValidator(t), + &mockStoredReqFetcher{}, + empty_fetcher.EmptyFetcher{}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{MaxRequestSize: maxSize}, + metrics, + analyticsConf.NewPBSAnalytics(&config.Analytics{}), + map[string]string{}, + false, + []byte{}, + openrtb_ext.BidderMap, + nil, + nil, + hardcodedResponseIPValidator{response: true}, + } for i, requestData := range testStoredRequests { - newRequest, errList := edep.processStoredRequests(context.Background(), json.RawMessage(requestData)) + newRequest, errList := deps.processStoredRequests(context.Background(), json.RawMessage(requestData)) if len(errList) != 0 { for _, err := range errList { if err != nil { @@ -640,6 +975,7 @@ func TestOversizedRequest(t *testing.T) { openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } req := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(reqBody)) @@ -674,6 +1010,7 @@ func TestRequestSizeEdgeCase(t *testing.T) { openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } req := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(reqBody)) @@ -756,130 +1093,425 @@ func TestExplicitAMP(t *testing.T) { return } - bidReq := openrtb.BidRequest{ - Site: &openrtb.Site{ - Ext: json.RawMessage(`{"amp":1}`), - }, + bidReq := openrtb.BidRequest{ + Site: &openrtb.Site{ + Ext: json.RawMessage(`{"amp":1}`), + }, + } + setSiteImplicitly(httpReq, &bidReq) + assert.JSONEq(t, `{"amp":1}`, string(bidReq.Site.Ext)) +} + +// TestContentType prevents #328 +func TestContentType(t *testing.T) { + endpoint, _ := NewEndpoint( + &mockExchange{}, + newParamsValidator(t), + &mockStoredReqFetcher{}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{MaxRequestSize: maxSize}, + pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), + analyticsConf.NewPBSAnalytics(&config.Analytics{}), + map[string]string{}, + []byte{}, + openrtb_ext.BidderMap, + ) + request := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, "site.json"))) + recorder := httptest.NewRecorder() + endpoint(recorder, request, nil) + + if recorder.Header().Get("Content-Type") != "application/json" { + t.Errorf("Content-Type should be application/json. Got %s", recorder.Header().Get("Content-Type")) + } +} + +func TestValidateImpExt(t *testing.T) { + type testCase struct { + description string + impExt json.RawMessage + expectedImpExt string + expectedErrs []error + } + testGroups := []struct { + description string + testCases []testCase + }{ + { + "Empty", + []testCase{ + { + description: "Empty", + impExt: nil, + expectedImpExt: "", + expectedErrs: []error{errors.New("request.imp[0].ext is required")}, + }, + }, + }, + { + "Unknown bidder tests", + []testCase{ + { + description: "Unknown Bidder only", + impExt: json.RawMessage(`{"unknownbidder":{"placement_id":555}}`), + expectedImpExt: `{"unknownbidder":{"placement_id":555}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Unknown Prebid Ext Bidder only", + impExt: json.RawMessage(`{"prebid":{"bidder":{"unknownbidder":{"placement_id":555}}}}`), + expectedImpExt: `{"prebid":{"bidder":{"unknownbidder":{"placement_id":555}}}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Unknown Prebid Ext Bidder + First Party Data Context", + impExt: json.RawMessage(`{"prebid":{"bidder":{"unknownbidder":{"placement_id":555}}},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"prebid":{"bidder":{"unknownbidder":{"placement_id":555}}},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Unknown Bidder + First Party Data Context", + impExt: json.RawMessage(`{"unknownbidder":{"placement_id":555} ,"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"unknownbidder":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Unknown Bidder + Disabled Bidder", + impExt: json.RawMessage(`{"unknownbidder":{"placement_id":555},"disabledbidder":{"foo":"bar"}}`), + expectedImpExt: `{"unknownbidder":{"placement_id":555},"disabledbidder":{"foo":"bar"}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Unknown Bidder + Disabled Prebid Ext Bidder", + impExt: json.RawMessage(`{"unknownbidder":{"placement_id":555},"prebid":{"bidder":{"disabledbidder":{"foo":"bar"}}}}`), + expectedImpExt: `{"unknownbidder":{"placement_id":555},"prebid":{"bidder":{"disabledbidder":{"foo":"bar"}}}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + }, + }, + { + "Disabled bidder tests", + []testCase{ + { + description: "Disabled Bidder", + impExt: json.RawMessage(`{"disabledbidder":{"foo":"bar"}}`), + expectedImpExt: `{}`, + expectedErrs: []error{ + &errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}, + errors.New("request.imp[0].ext must contain at least one bidder"), + }, + // if only bidder(s) found in request.imp[x].ext.{biddername} or request.imp[x].ext.prebid.bidder.{biddername} are disabled, return error + }, + { + description: "Disabled Prebid Ext Bidder", + impExt: json.RawMessage(`{"prebid":{"bidder":{"disabledbidder":{"foo":"bar"}}}}`), + expectedImpExt: `{"prebid":{"bidder":{"disabledbidder":{"foo":"bar"}}}}`, + expectedErrs: []error{ + &errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}, + errors.New("request.imp[0].ext must contain at least one bidder"), + }, + }, + { + description: "Disabled Bidder + First Party Data Context", + impExt: json.RawMessage(`{"disabledbidder":{"foo":"bar"},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{ + &errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}, + errors.New("request.imp[0].ext must contain at least one bidder"), + }, + }, + { + description: "Disabled Prebid Ext Bidder + First Party Data Context", + impExt: json.RawMessage(`{"prebid":{"bidder":{"disabledbidder":{"foo":"bar"}}},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"context":{"data":{"keywords":"prebid server example"}}, "prebid":{"bidder":{"disabledbidder":{"foo":"bar"}}}}`, + expectedErrs: []error{ + &errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}, + errors.New("request.imp[0].ext must contain at least one bidder"), + }, + }, + }, + }, + { + "First Party only", + []testCase{ + { + description: "First Party Data Context", + impExt: json.RawMessage(`{"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{ + errors.New("request.imp[0].ext must contain at least one bidder"), + }, + }, + }, + }, + { + "Valid bidder tests", + []testCase{ + { + description: "Valid bidder root ext", + impExt: json.RawMessage(`{"appnexus":{"placement_id":555}}`), + expectedImpExt: `{"appnexus":{"placement_id":555}}`, + expectedErrs: []error{}, + }, + { + description: "Valid bidder in prebid field", + impExt: json.RawMessage(`{"prebid":{"bidder":{"appnexus":{"placement_id":555}}}}`), + expectedImpExt: `{"prebid":{"bidder":{"appnexus":{"placement_id":555}}}}`, + expectedErrs: []error{}, + }, + { + description: "Valid Bidder + First Party Data Context", + impExt: json.RawMessage(`{"appnexus":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"appnexus":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{}, + }, + { + description: "Valid Prebid Ext Bidder + First Party Data Context", + impExt: json.RawMessage(`{"prebid":{"bidder":{"appnexus":{"placement_id":555}}} ,"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"prebid":{"bidder":{"appnexus":{"placement_id":555}}},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{}, + }, + { + description: "Valid Bidder + Unknown Bidder", + impExt: json.RawMessage(`{"appnexus":{"placement_id":555},"unknownbidder":{"placement_id":555}}`), + expectedImpExt: `{"appnexus":{"placement_id":555},"unknownbidder":{"placement_id":555}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Valid Bidder + Disabled Bidder", + impExt: json.RawMessage(`{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"}}`), + expectedImpExt: `{"appnexus":{"placement_id":555}}`, + expectedErrs: []error{&errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}}, + }, + { + description: "Valid Bidder + Disabled Bidder + First Party Data Context", + impExt: json.RawMessage(`{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"appnexus":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{&errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}}, + }, + { + description: "Valid Bidder + Disabled Bidder + Unknown Bidder + First Party Data Context", + impExt: json.RawMessage(`{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"},"unknownbidder":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"},"unknownbidder":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + { + description: "Valid Prebid Ext Bidder + Disabled Bidder Ext", + impExt: json.RawMessage(`{"prebid":{"bidder":{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"}}}}`), + expectedImpExt: `{"prebid":{"bidder":{"appnexus":{"placement_id": 555},"disabledbidder":{"foo":"bar"}}},"appnexus":{"placement_id":555}}`, + expectedErrs: []error{&errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}}, + }, + { + description: "Valid Prebid Ext Bidder + Disabled Ext Bidder + First Party Data Context", + impExt: json.RawMessage(`{"prebid":{"bidder":{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"}}},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"prebid":{"bidder":{"appnexus":{"placement_id": 555},"disabledbidder":{"foo":"bar"}}},"appnexus":{"placement_id":555},"context":{"data":{"keywords":"prebid server example"}}}`, + expectedErrs: []error{&errortypes.BidderTemporarilyDisabled{Message: "The bidder 'disabledbidder' has been disabled."}}, + }, + { + description: "Valid Prebid Ext Bidder + Disabled Ext Bidder + Unknown Ext + First Party Data Context", + impExt: json.RawMessage(`{"prebid":{"bidder":{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"},"unknownbidder":{"placement_id":555}}},"context":{"data":{"keywords":"prebid server example"}}}`), + expectedImpExt: `{"context":{"data":{"keywords":"prebid server example"}},"prebid":{"bidder":{"appnexus":{"placement_id":555},"disabledbidder":{"foo":"bar"},"unknownbidder":{"placement_id":555}}}}`, + expectedErrs: []error{errors.New("request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?")}, + }, + }, + }, + } + + deps := &endpointDeps{ + &nobidExchange{}, + newParamsValidator(t), + &mockStoredReqFetcher{}, + empty_fetcher.EmptyFetcher{}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{MaxRequestSize: int64(8096)}, + pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), + analyticsConf.NewPBSAnalytics(&config.Analytics{}), + map[string]string{"disabledbidder": "The bidder 'disabledbidder' has been disabled."}, + false, + []byte{}, + openrtb_ext.BidderMap, + nil, + nil, + hardcodedResponseIPValidator{response: true}, + } + + for _, group := range testGroups { + for _, test := range group.testCases { + imp := &openrtb.Imp{Ext: test.impExt} + + errs := deps.validateImpExt(imp, nil, 0) + + if len(test.expectedImpExt) > 0 { + assert.JSONEq(t, test.expectedImpExt, string(imp.Ext), "imp.ext JSON does not match expected. Test: %s. %s\n", group.description, test.description) + } else { + assert.Empty(t, imp.Ext, "imp.ext expected to be empty but was: %s. Test: %s. %s\n", string(imp.Ext), group.description, test.description) + } + assert.Equal(t, test.expectedErrs, errs, "errs slice does not match expected. Test: %s. %s\n", group.description, test.description) + } + } +} + +func validRequest(t *testing.T, filename string) string { + requestData, err := ioutil.ReadFile("sample-requests/valid-whole/supplementary/" + filename) + if err != nil { + t.Fatalf("Failed to fetch a valid request: %v", err) } - setSiteImplicitly(httpReq, &bidReq) - assert.JSONEq(t, `{"amp":1}`, string(bidReq.Site.Ext)) + testBidRequest, _, _, err := jsonparser.Get(requestData, "mockBidRequest") + assert.NoError(t, err, "Error jsonparsing root.mockBidRequest from file %s. Desc: %v.", filename, err) + + return string(testBidRequest) } -// TestContentType prevents #328 -func TestContentType(t *testing.T) { - endpoint, _ := NewEndpoint( - &mockExchange{}, +func TestCurrencyTrunc(t *testing.T) { + deps := &endpointDeps{ + &nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, - &config.Configuration{MaxRequestSize: maxSize}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{}, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, + false, []byte{}, openrtb_ext.BidderMap, - ) - request := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(validRequest(t, "site.json"))) - recorder := httptest.NewRecorder() - endpoint(recorder, request, nil) - - if recorder.Header().Get("Content-Type") != "application/json" { - t.Errorf("Content-Type should be application/json. Got %s", recorder.Header().Get("Content-Type")) + nil, + nil, + hardcodedResponseIPValidator{response: true}, } -} -// TestDisabledBidder makes sure we pass when encountering a disabled bidder in the configuration. -func TestDisabledBidder(t *testing.T) { - reqData, err := ioutil.ReadFile("sample-requests/invalid-whole/unknown-bidder.json") - if err != nil { - t.Fatalf("Failed to fetch a valid request: %v", err) + ui := uint64(1) + req := openrtb.BidRequest{ + ID: "someID", + Imp: []openrtb.Imp{ + { + ID: "imp-ID", + Banner: &openrtb.Banner{ + W: &ui, + H: &ui, + }, + Ext: json.RawMessage("{\"appnexus\": {\"placementId\": 5667}}"), + }, + }, + Site: &openrtb.Site{ + ID: "myID", + }, + Cur: []string{"USD", "EUR"}, } - reqBody := string(getRequestPayload(t, reqData)) + errL := deps.validateRequest(&req) + + expectedError := errortypes.Warning{Message: "A prebid request can only process one currency. Taking the first currency in the list, USD, as the active currency"} + assert.ElementsMatch(t, errL, []error{&expectedError}) +} + +func TestCCPAInvalid(t *testing.T) { deps := &endpointDeps{ &nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, - &config.Configuration{ - MaxRequestSize: int64(len(reqBody)), - }, + &config.Configuration{}, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), analyticsConf.NewPBSAnalytics(&config.Analytics{}), - map[string]string{"unknownbidder": "The bidder 'unknownbidder' has been disabled."}, + map[string]string{}, false, []byte{}, openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } - req := httptest.NewRequest("POST", "/openrtb2/auction", strings.NewReader(reqBody)) - recorder := httptest.NewRecorder() + ui := uint64(1) + req := openrtb.BidRequest{ + ID: "someID", + Imp: []openrtb.Imp{ + { + ID: "imp-ID", + Banner: &openrtb.Banner{ + W: &ui, + H: &ui, + }, + Ext: json.RawMessage(`{"appnexus": {"placementId": 5667}}`), + }, + }, + Site: &openrtb.Site{ + ID: "myID", + }, + Regs: &openrtb.Regs{ + Ext: json.RawMessage(`{"us_privacy":"invalid by length"}`), + }, + } - deps.Auction(recorder, req, nil) + errL := deps.validateRequest(&req) - if recorder.Code != http.StatusOK { - t.Errorf("Endpoint should return a 200 if the unknown bidder was disabled.") - } + expectedWarning := errortypes.InvalidPrivacyConsent{Message: "CCPA consent is invalid and will be ignored. (request.regs.ext.us_privacy must contain 4 characters)"} + assert.ElementsMatch(t, errL, []error{&expectedWarning}) - if bytesRead, err := req.Body.Read(make([]byte, 1)); bytesRead != 0 || err != io.EOF { - t.Errorf("The request body should have been read to completion.") - } + assert.Empty(t, req.Regs.Ext, "Invalid Consent Removed From Request") } -func TestValidateImpExtDisabledBidder(t *testing.T) { - imp := &openrtb.Imp{ - Ext: json.RawMessage(`{"appnexus":{"placement_id":555},"unknownbidder":{"foo":"bar"}}`), - } +func TestNoSaleInvalid(t *testing.T) { deps := &endpointDeps{ &nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, - &config.Configuration{MaxRequestSize: int64(8096)}, + &config.Configuration{}, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), analyticsConf.NewPBSAnalytics(&config.Analytics{}), - map[string]string{"unknownbidder": "The bidder 'unknownbidder' has been disabled."}, + map[string]string{}, false, []byte{}, openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } - errs := deps.validateImpExt(imp, nil, 0) - assert.JSONEq(t, `{"appnexus":{"placement_id":555}}`, string(imp.Ext)) - assert.Equal(t, []error{&errortypes.BidderTemporarilyDisabled{Message: "The bidder 'unknownbidder' has been disabled."}}, errs) -} -func TestEffectivePubID(t *testing.T) { - var pub openrtb.Publisher - assert.Equal(t, pbsmetrics.PublisherUnknown, effectivePubID(nil), "effectivePubID failed for nil Publisher.") - assert.Equal(t, pbsmetrics.PublisherUnknown, effectivePubID(&pub), "effectivePubID failed for empty Publisher.") - pub.ID = "123" - assert.Equal(t, "123", effectivePubID(&pub), "effectivePubID failed for standard Publisher.") - pub.Ext = json.RawMessage(`{"prebid": {"parentAccount": "abc"} }`) - assert.Equal(t, "abc", effectivePubID(&pub), "effectivePubID failed for parentAccount.") + ui := uint64(1) + req := openrtb.BidRequest{ + ID: "someID", + Imp: []openrtb.Imp{ + { + ID: "imp-ID", + Banner: &openrtb.Banner{ + W: &ui, + H: &ui, + }, + Ext: json.RawMessage(`{"appnexus": {"placementId": 5667}}`), + }, + }, + Site: &openrtb.Site{ + ID: "myID", + }, + Regs: &openrtb.Regs{ + Ext: json.RawMessage(`{"us_privacy":"1NYN"}`), + }, + Ext: json.RawMessage(`{"prebid":{"nosale":["*", "appnexus"]}}`), + } + + errL := deps.validateRequest(&req) + + expectedError := errors.New("request.ext.prebid.nosale is invalid: can only specify all bidders if no other bidders are provided") + assert.ElementsMatch(t, errL, []error{expectedError}) } -func validRequest(t *testing.T, filename string) string { - requestData, err := ioutil.ReadFile("sample-requests/valid-whole/supplementary/" + filename) - if err != nil { - t.Fatalf("Failed to fetch a valid request: %v", err) +func TestValidateSourceTID(t *testing.T) { + cfg := &config.Configuration{ + AutoGenSourceTID: true, } - return string(requestData) -} -func TestCurrencyTrunc(t *testing.T) { deps := &endpointDeps{ &nobidExchange{}, newParamsValidator(t), &mockStoredReqFetcher{}, empty_fetcher.EmptyFetcher{}, empty_fetcher.EmptyFetcher{}, - &config.Configuration{}, + cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), analyticsConf.NewPBSAnalytics(&config.Analytics{}), map[string]string{}, @@ -888,6 +1520,7 @@ func TestCurrencyTrunc(t *testing.T) { openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } ui := uint64(1) @@ -900,22 +1533,22 @@ func TestCurrencyTrunc(t *testing.T) { W: &ui, H: &ui, }, - Ext: json.RawMessage("{\"appnexus\": {\"placementId\": 5667}}"), + Ext: json.RawMessage(`{"appnexus": {"placementId": 5667}}`), }, }, Site: &openrtb.Site{ ID: "myID", }, - Cur: []string{"USD", "EUR"}, + Regs: &openrtb.Regs{ + Ext: json.RawMessage(`{"us_privacy":"invalid by length"}`), + }, } - errL := deps.validateRequest(&req) - - expectedError := errortypes.Warning{Message: "A prebid request can only process one currency. Taking the first currency in the list, USD, as the active currency"} - assert.ElementsMatch(t, errL, []error{&expectedError}) + deps.validateRequest(&req) + assert.NotEmpty(t, req.Source.TID, "Expected req.Source.TID to be filled with a randomly generated UID") } -func TestCCPAInvalid(t *testing.T) { +func TestSChainInvalid(t *testing.T) { deps := &endpointDeps{ &nobidExchange{}, newParamsValidator(t), @@ -931,6 +1564,7 @@ func TestCCPAInvalid(t *testing.T) { openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } ui := uint64(1) @@ -950,16 +1584,190 @@ func TestCCPAInvalid(t *testing.T) { ID: "myID", }, Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"invalid by length"}`), + Ext: json.RawMessage(`{"us_privacy":"abcd"}`), }, + Ext: json.RawMessage(`{"prebid":{"schains":[{"bidders":["appnexus"],"schain":{"complete":1,"nodes":[{"asi":"directseller1.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}, {"bidders":["appnexus"],"schain":{"complete":1,"nodes":[{"asi":"directseller2.com","sid":"00002","rid":"BidRequest2","hp":1}],"ver":"1.0"}}]}}`), } errL := deps.validateRequest(&req) - expectedWarning := errortypes.InvalidPrivacyConsent{Message: "CCPA consent is invalid and will be ignored. (request.regs.ext.us_privacy must contain 4 characters)"} - assert.ElementsMatch(t, errL, []error{&expectedWarning}) + expectedError := fmt.Errorf("request.ext.prebid.schains contains multiple schains for bidder appnexus; it must contain no more than one per bidder.") + assert.ElementsMatch(t, errL, []error{expectedError}) +} - assert.Empty(t, req.Regs.Ext, "Invalid Consent Removed From Request") +func TestGetAccountID(t *testing.T) { + testPubID := "test-pub" + testParentAccount := "test-account" + testPubExt := openrtb_ext.ExtPublisher{ + Prebid: &openrtb_ext.ExtPublisherPrebid{ + ParentAccount: &testParentAccount, + }, + } + testPubExtJSON, err := json.Marshal(testPubExt) + assert.NoError(t, err) + + testCases := []struct { + description string + pub *openrtb.Publisher + expectedAccID string + }{ + { + description: "Publisher.ID and Publisher.Ext.Prebid.ParentAccount both present", + pub: &openrtb.Publisher{ + ID: testPubID, + Ext: testPubExtJSON, + }, + expectedAccID: testParentAccount, + }, + { + description: "Only Publisher.Ext.Prebid.ParentAccount present", + pub: &openrtb.Publisher{ + ID: "", + Ext: testPubExtJSON, + }, + expectedAccID: testParentAccount, + }, + { + description: "Only Publisher.ID present", + pub: &openrtb.Publisher{ + ID: testPubID, + }, + expectedAccID: testPubID, + }, + { + description: "Neither Publisher.ID or Publisher.Ext.Prebid.ParentAccount present", + pub: &openrtb.Publisher{}, + expectedAccID: pbsmetrics.PublisherUnknown, + }, + { + description: "Publisher is nil", + pub: nil, + expectedAccID: pbsmetrics.PublisherUnknown, + }, + } + + for _, test := range testCases { + acc := getAccountID(test.pub) + assert.Equal(t, test.expectedAccID, acc, "getAccountID should return expected account for test case: %s", test.description) + } +} + +func TestSanitizeRequest(t *testing.T) { + testCases := []struct { + description string + req *openrtb.BidRequest + ipValidator iputil.IPValidator + expectedIPv4 string + expectedIPv6 string + }{ + { + description: "Empty", + req: &openrtb.BidRequest{ + Device: &openrtb.Device{ + IP: "", + IPv6: "", + }, + }, + expectedIPv4: "", + expectedIPv6: "", + }, + { + description: "Valid", + req: &openrtb.BidRequest{ + Device: &openrtb.Device{ + IP: "1.1.1.1", + IPv6: "1111::", + }, + }, + ipValidator: hardcodedResponseIPValidator{response: true}, + expectedIPv4: "1.1.1.1", + expectedIPv6: "1111::", + }, + { + description: "Invalid", + req: &openrtb.BidRequest{ + Device: &openrtb.Device{ + IP: "1.1.1.1", + IPv6: "1111::", + }, + }, + ipValidator: hardcodedResponseIPValidator{response: false}, + expectedIPv4: "", + expectedIPv6: "", + }, + { + description: "Invalid - Wrong IP Types", + req: &openrtb.BidRequest{ + Device: &openrtb.Device{ + IP: "1111::", + IPv6: "1.1.1.1", + }, + }, + ipValidator: hardcodedResponseIPValidator{response: true}, + expectedIPv4: "", + expectedIPv6: "", + }, + { + description: "Malformed", + req: &openrtb.BidRequest{ + Device: &openrtb.Device{ + IP: "malformed", + IPv6: "malformed", + }, + }, + expectedIPv4: "", + expectedIPv6: "", + }, + } + + for _, test := range testCases { + sanitizeRequest(test.req, test.ipValidator) + assert.Equal(t, test.expectedIPv4, test.req.Device.IP, test.description+":ipv4") + assert.Equal(t, test.expectedIPv6, test.req.Device.IPv6, test.description+":ipv6") + } +} + +func TestValidateAndFillSourceTID(t *testing.T) { + testTID := "some-tid" + testCases := []struct { + description string + req *openrtb.BidRequest + expectRandTID bool + expectedTID string + }{ + { + description: "req.Source not present. Expecting a randomly generated TID value", + req: &openrtb.BidRequest{}, + expectRandTID: true, + }, + { + description: "req.Source.TID not present. Expecting a randomly generated TID value", + req: &openrtb.BidRequest{ + Source: &openrtb.Source{}, + }, + expectRandTID: true, + }, + { + description: "req.Source.TID present. Expecting no change", + req: &openrtb.BidRequest{ + Source: &openrtb.Source{ + TID: testTID, + }, + }, + expectRandTID: false, + expectedTID: testTID, + }, + } + + for _, test := range testCases { + _ = validateAndFillSourceTID(test.req) + if test.expectRandTID { + assert.NotEmpty(t, test.req.Source.TID, test.description) + assert.NotEqual(t, test.expectedTID, test.req.Source.TID, test.description) + } else { + assert.Equal(t, test.expectedTID, test.req.Source.TID, test.description) + } + } } // nobidExchange is a well-behaved exchange which always bids "no bid". @@ -967,28 +1775,70 @@ type nobidExchange struct { gotRequest *openrtb.BidRequest } -func (e *nobidExchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, ids exchange.IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { - e.gotRequest = bidRequest +func (e *nobidExchange) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + e.gotRequest = r.BidRequest return &openrtb.BidResponse{ - ID: bidRequest.ID, + ID: r.BidRequest.ID, BidID: "test bid id", NBR: openrtb.NoBidReasonCodeUnknownError.Ptr(), }, nil } -type brokenExchange struct{} - -func (e *brokenExchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, ids exchange.IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { - return nil, errors.New("Critical, unrecoverable error.") +type mockBidExchange struct { + gotRequest *openrtb.BidRequest } -func getMessage(t *testing.T, example []byte) []byte { - if value, err := jsonparser.GetString(example, "message"); err != nil { - t.Fatalf("Error parsing root.message from request: %v.", err) - } else { - return []byte(value) +// mockBidExchange is a well-behaved exchange that lists the bidders found in every bidRequest.Imp[i].Ext +// into the bidResponse.Ext to assert the bidder adapters that were not filtered out in the validation process +func (e *mockBidExchange) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + bidResponse := &openrtb.BidResponse{ + ID: r.BidRequest.ID, + BidID: "test bid id", + NBR: openrtb.NoBidReasonCodeUnknownError.Ptr(), } - return nil + if len(r.BidRequest.Imp) > 0 { + var SeatBidMap = make(map[string]openrtb.SeatBid, 0) + for _, imp := range r.BidRequest.Imp { + var bidderExts map[string]json.RawMessage + if err := json.Unmarshal(imp.Ext, &bidderExts); err != nil { + return nil, err + } + + if rawPrebidExt, ok := bidderExts[openrtb_ext.PrebidExtKey]; ok { + var prebidExt openrtb_ext.ExtImpPrebid + if err := json.Unmarshal(rawPrebidExt, &prebidExt); err == nil && prebidExt.Bidder != nil { + for bidder, ext := range prebidExt.Bidder { + if ext == nil { + continue + } + + bidderExts[bidder] = ext + } + } + } + + for bidderNameOrAlias := range bidderExts { + if isBidderToValidate(bidderNameOrAlias) { + if val, ok := SeatBidMap[bidderNameOrAlias]; ok { + val.Bid = append(val.Bid, openrtb.Bid{ID: fmt.Sprintf("%s-bid", bidderNameOrAlias)}) + } else { + SeatBidMap[bidderNameOrAlias] = openrtb.SeatBid{Seat: fmt.Sprintf("%s-bids", bidderNameOrAlias), Bid: []openrtb.Bid{{ID: fmt.Sprintf("%s-bid", bidderNameOrAlias)}}} + } + } + } + } + for _, seatBid := range SeatBidMap { + bidResponse.SeatBid = append(bidResponse.SeatBid, seatBid) + } + } + + return bidResponse, nil +} + +type brokenExchange struct{} + +func (e *brokenExchange) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + return nil, errors.New("Critical, unrecoverable error.") } // StoredRequest testing @@ -1000,7 +1850,7 @@ func getMessage(t *testing.T, example []byte) []byte { // second below is identical to first but with extra '}' for invalid JSON var testStoredRequestData = map[string]json.RawMessage{ "2": json.RawMessage(`{ - "tmax": 500, +"tmax": 500, "ext": { "prebid": { "targeting": { @@ -1010,15 +1860,15 @@ var testStoredRequestData = map[string]json.RawMessage{ } }`), "3": json.RawMessage(`{ - "tmax": 500, - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - } - } - }} - }`), +"tmax": 500, + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + } + } + }} + }`), } // Stored Imp Requests @@ -1027,7 +1877,7 @@ var testStoredRequestData = map[string]json.RawMessage{ // third below has valid JSON and matches schema var testStoredImpData = map[string]json.RawMessage{ "1": json.RawMessage(`{ - "id": "adUnit1", +"id": "adUnit1", "ext": { "appnexus": { "placementId": "abc", @@ -1040,7 +1890,7 @@ var testStoredImpData = map[string]json.RawMessage{ } }`), "7": json.RawMessage(`{ - "id": "adUnit1", +"id": "adUnit1", "ext": { "appnexus": { "placementId": 12345678, @@ -1055,7 +1905,7 @@ var testStoredImpData = map[string]json.RawMessage{ } }`), "9": json.RawMessage(`{ - "id": "adUnit1", +"id": "adUnit1", "ext": { "appnexus": { "placementId": 12345678, @@ -1334,12 +2184,27 @@ func (cf mockStoredReqFetcher) FetchRequests(ctx context.Context, requestIDs []s return testStoredRequestData, testStoredImpData, nil } +var mockAccountData = map[string]json.RawMessage{ + "valid_acct": json.RawMessage(`{"disabled":false}`), +} + +type mockAccountFetcher struct { +} + +func (af mockAccountFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + if account, ok := mockAccountData[accountID]; ok { + return account, nil + } else { + return nil, []error{stored_requests.NotFoundError{accountID, "Account"}} + } +} + type mockExchange struct { lastRequest *openrtb.BidRequest } -func (m *mockExchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, ids exchange.IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { - m.lastRequest = bidRequest +func (m *mockExchange) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + m.lastRequest = r.BidRequest return &openrtb.BidResponse{ SeatBid: []openrtb.SeatBid{{ Bid: []openrtb.Bid{{ @@ -1349,20 +2214,6 @@ func (m *mockExchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidR }, nil } -func blankAdapterConfig(bidderList []openrtb_ext.BidderName, disabledBidders []string) map[string]config.Adapter { - adapters := make(map[string]config.Adapter) - for _, b := range bidderList { - adapters[string(b)] = config.Adapter{} - } - for _, b := range disabledBidders { - tmp := adapters[b] - tmp.Disabled = true - adapters[b] = tmp - } - - return adapters -} - func getBidderInfos(cfg map[string]config.Adapter, biddersNames []openrtb_ext.BidderName) adapters.BidderInfos { biddersInfos := make(adapters.BidderInfos) for _, name := range biddersNames { @@ -1385,3 +2236,11 @@ func newBidderInfo(cfg config.Adapter) adapters.BidderInfo { Status: status, } } + +type hardcodedResponseIPValidator struct { + response bool +} + +func (v hardcodedResponseIPValidator) IsValid(net.IP, iputil.IPVersion) bool { + return v.response +} diff --git a/endpoints/openrtb2/ctv_auction.go b/endpoints/openrtb2/ctv_auction.go index a0ea8214510..551257c2599 100644 --- a/endpoints/openrtb2/ctv_auction.go +++ b/endpoints/openrtb2/ctv_auction.go @@ -14,6 +14,7 @@ import ( "github.com/PubMatic-OpenWrap/etree" "github.com/PubMatic-OpenWrap/openrtb" + accountService "github.com/PubMatic-OpenWrap/prebid-server/account" "github.com/PubMatic-OpenWrap/prebid-server/analytics" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/endpoints/openrtb2/ctv/combination" @@ -28,6 +29,7 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/usersync" + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" "github.com/buger/jsonparser" uuid "github.com/gofrs/uuid" "github.com/golang/glog" @@ -55,7 +57,8 @@ func NewCTVEndpoint( validator openrtb_ext.BidderParamValidator, requestsByID stored_requests.Fetcher, videoFetcher stored_requests.Fetcher, - categories stored_requests.CategoryFetcher, + accounts stored_requests.AccountFetcher, + //categories stored_requests.CategoryFetcher, cfg *config.Configuration, met pbsmetrics.MetricsEngine, pbsAnalytics analytics.PBSAnalyticsModule, @@ -63,18 +66,23 @@ func NewCTVEndpoint( defReqJSON []byte, bidderMap map[string]openrtb_ext.BidderName) (httprouter.Handle, error) { - if ex == nil || validator == nil || requestsByID == nil || cfg == nil || met == nil { + if ex == nil || validator == nil || requestsByID == nil || accounts == nil || cfg == nil || met == nil { return nil, errors.New("NewCTVEndpoint requires non-nil arguments.") } defRequest := defReqJSON != nil && len(defReqJSON) > 0 + ipValidator := iputil.PublicNetworkIPValidator{ + IPv4PrivateNetworks: cfg.RequestValidation.IPv4PrivateNetworksParsed, + IPv6PrivateNetworks: cfg.RequestValidation.IPv6PrivateNetworksParsed, + } + return httprouter.Handle((&ctvEndpointDeps{ endpointDeps: endpointDeps{ ex, validator, requestsByID, videoFetcher, - categories, + accounts, cfg, met, pbsAnalytics, @@ -84,6 +92,7 @@ func NewCTVEndpoint( bidderMap, nil, nil, + ipValidator, }, }).CTVAuctionEndpoint), nil } @@ -157,7 +166,7 @@ func (deps *ctvEndpointDeps) CTVAuctionEndpoint(w http.ResponseWriter, r *http.R if request.App != nil { deps.labels.Source = pbsmetrics.DemandApp deps.labels.RType = pbsmetrics.ReqTypeVideo - deps.labels.PubID = effectivePubID(request.App.Publisher) + deps.labels.PubID = getAccountID(request.App.Publisher) } else { //request.Site != nil deps.labels.Source = pbsmetrics.DemandWeb if usersyncs.LiveSyncCount() == 0 { @@ -165,18 +174,19 @@ func (deps *ctvEndpointDeps) CTVAuctionEndpoint(w http.ResponseWriter, r *http.R } else { deps.labels.CookieFlag = pbsmetrics.CookieFlagYes } - deps.labels.PubID = effectivePubID(request.Site.Publisher) + deps.labels.PubID = getAccountID(request.Site.Publisher) } - //Validate Accounts - if err = validateAccount(deps.cfg, deps.labels.PubID); err != nil { - errL = append(errL, err) + deps.ctx = context.Background() + + // Look up account now that we have resolved the pubID value + account, acctIDErrs := accountService.GetAccount(deps.ctx, deps.cfg, deps.accounts, deps.labels.PubID) + if len(acctIDErrs) > 0 { + errL = append(errL, acctIDErrs...) writeError(errL, w, &deps.labels) return } - deps.ctx = context.Background() - //Setting Timeout for Request timeout := deps.cfg.AuctionTimeouts.LimitAuctionTimeout(time.Duration(request.TMax) * time.Millisecond) if timeout > 0 { @@ -185,7 +195,8 @@ func (deps *ctvEndpointDeps) CTVAuctionEndpoint(w http.ResponseWriter, r *http.R defer cancel() } - response, err = deps.holdAuction(request, usersyncs) + response, err = deps.holdAuction(request, usersyncs, account) + ao.Request = request ao.Response = response if err != nil || nil == response { @@ -234,7 +245,7 @@ func (deps *ctvEndpointDeps) CTVAuctionEndpoint(w http.ResponseWriter, r *http.R } } -func (deps *ctvEndpointDeps) holdAuction(request *openrtb.BidRequest, usersyncs *usersync.PBSCookie) (*openrtb.BidResponse, error) { +func (deps *ctvEndpointDeps) holdAuction(request *openrtb.BidRequest, usersyncs *usersync.PBSCookie, account *config.Account) (*openrtb.BidResponse, error) { defer util.TimeTrack(time.Now(), fmt.Sprintf("Tid:%v CTVHoldAuction", deps.request.ID)) //Hold OpenRTB Standard Auction @@ -243,7 +254,15 @@ func (deps *ctvEndpointDeps) holdAuction(request *openrtb.BidRequest, usersyncs return &openrtb.BidResponse{ID: request.ID}, nil } - return deps.ex.HoldAuction(deps.ctx, request, usersyncs, deps.labels, &deps.categories, nil) + auctionRequest := exchange.AuctionRequest{ + BidRequest: request, + Account: *account, + UserSyncs: usersyncs, + RequestType: deps.labels.RType, + LegacyLabels: deps.labels, + } + + return deps.ex.HoldAuction(deps.ctx, auctionRequest, nil) } /********************* BidRequest Processing *********************/ diff --git a/endpoints/openrtb2/sample-requests/account-required/no-account/not-required-no-acct.json b/endpoints/openrtb2/sample-requests/account-required/no-account/not-required-no-acct.json new file mode 100644 index 00000000000..c3ab09d4883 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/account-required/no-account/not-required-no-acct.json @@ -0,0 +1,84 @@ +{ + "description": "This request comes with no account id and the mock config does not make it a requirement", + "config": { + "accountRequired": false + }, + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "user": { }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "tmax": 500, + "ext": { + "prebid": { + "aliases": { + "districtm": "appnexus" + }, + "bidadjustmentfactors": { + "appnexus": 1.01, + "districtm": 0.98, + "rubicon": 0.99 + }, + "cache": { + "bids": {} + }, + "targeting": { + "includewinners": false, + "pricegranularity": { + "precision": 2, + "ranges": [ + { + "max": 20, + "increment": 0.10 + } + ] + } + } + } + } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/account-required/no-account/required-no-acct.json b/endpoints/openrtb2/sample-requests/account-required/no-account/required-no-acct.json new file mode 100644 index 00000000000..f6c91918f13 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/account-required/no-account/required-no-acct.json @@ -0,0 +1,68 @@ +{ + "description": "This request comes with no account id and the mock config requires it. We expect an error", + "config": { + "accountRequired": true + }, + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "user": { }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "tmax": 500, + "ext": { + "prebid": { + "aliases": { + "districtm": "appnexus" + }, + "bidadjustmentfactors": { + "appnexus": 1.01, + "districtm": 0.98, + "rubicon": 0.99 + }, + "cache": { + "bids": {} + }, + "targeting": { + "includewinners": false, + "pricegranularity": { + "precision": 2, + "ranges": [ + { + "max": 20, + "increment": 0.10 + } + ] + } + } + } + } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Prebid-server has been configured to discard requests without a valid Account ID. Please reach out to the prebid server host.\n" +} diff --git a/endpoints/openrtb2/sample-requests/account-required/no-acct.json b/endpoints/openrtb2/sample-requests/account-required/no-acct.json deleted file mode 100644 index d84d797017d..00000000000 --- a/endpoints/openrtb2/sample-requests/account-required/no-acct.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "description": "This request comes with no account id", - "message": "Invalid request: Prebid-server has been configured to discard requests that don't come with an Account ID. Please reach out to the prebid server host.\n", - - "requestPayload": { - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "user": { }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "tmax": 500, - "ext": { - "prebid": { - "aliases": { - "districtm": "appnexus" - }, - "bidadjustmentfactors": { - "appnexus": 1.01, - "districtm": 0.98, - "rubicon": 0.99 - }, - "cache": { - "bids": {} - }, - "targeting": { - "includewinners": false, - "pricegranularity": { - "precision": 2, - "ranges": [ - { - "max": 20, - "increment": 0.10 - } - ] - } - } - } - } - } - } - diff --git a/endpoints/openrtb2/sample-requests/account-required/with-acct.json b/endpoints/openrtb2/sample-requests/account-required/valid-acct.json similarity index 79% rename from endpoints/openrtb2/sample-requests/account-required/with-acct.json rename to endpoints/openrtb2/sample-requests/account-required/valid-acct.json index fb4c6313051..15e72323c8e 100644 --- a/endpoints/openrtb2/sample-requests/account-required/with-acct.json +++ b/endpoints/openrtb2/sample-requests/account-required/valid-acct.json @@ -1,12 +1,12 @@ { - "description": "This request comes with no account id", - "message": "Invalid request: Prebid-server has been configured to discard requests that don't come with an Account ID. Please reach out to the prebid server host.\n", - + "description": "This request comes with a valid account id", + "message": "", + "requestPayload": { "id": "some-request-id", "site": { - "publisher": { "id": "not_bad_acct"}, - "page": "test.somepage.com" + "publisher": { "id": "valid_acct"}, + "page": "test.somepage.com" }, "user": { }, "imp": [ @@ -64,4 +64,4 @@ } } } - + diff --git a/endpoints/openrtb2/sample-requests/account-required/with-account/required-blacklisted-acct.json b/endpoints/openrtb2/sample-requests/account-required/with-account/required-blacklisted-acct.json new file mode 100644 index 00000000000..894a92fdb27 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/account-required/with-account/required-blacklisted-acct.json @@ -0,0 +1,91 @@ +{ + "description": "Account is required but request comes with a blacklisted account id", + "config": { + "accountRequired": true, + "blacklistedAccts": ["bad_acct"] + }, + "mockBidRequest": { + "id": "some-request-id", + "user": { + "ext": { + "consent": "gdpr-consent-string", + "prebid": { + "buyeruids": { + "appnexus": "override-appnexus-id-in-cookie" + } + } + } + }, + "app": { + "id": "cool_app", + "publisher": { + "id": "bad_acct" + } + }, + "regs": { + "ext": { + "gdpr": 1 + } + }, + "imp": [ + { + "id": "some-impression-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + }, + "districtm": { + "placementId": 105 + }, + "rubicon": { + "accountId": 1001, + "siteId": 113932, + "zoneId": 535510 + } + } + } + ], + "tmax": 500, + "ext": { + "prebid": { + "aliases": { + "districtm": "appnexus" + }, + "bidadjustmentfactors": { + "appnexus": 1.01, + "districtm": 0.98, + "rubicon": 0.99 + }, + "cache": { + "bids": {} + }, + "targeting": { + "includewinners": false, + "pricegranularity": { + "precision": 2, + "ranges": [ + { + "max": 20, + "increment": 0.10 + } + ] + } + } + } + } + }, + "expectedReturnCode": 503, + "expectedErrorMessage": "Invalid request: Prebid-server has disabled Account ID: bad_acct, please reach out to the prebid server host.\n" +} diff --git a/endpoints/openrtb2/sample-requests/account-required/with-account/required-with-acct.json b/endpoints/openrtb2/sample-requests/account-required/with-account/required-with-acct.json new file mode 100644 index 00000000000..a72d184c81c --- /dev/null +++ b/endpoints/openrtb2/sample-requests/account-required/with-account/required-with-acct.json @@ -0,0 +1,86 @@ +{ + "description": "This request comes with an account id and which is required by the config", + "config": { + "accountRequired": true + }, + + "mockBidRequest": { + "id": "some-request-id", + "site": { + "publisher": { "id": "not_bad_acct"}, + "page": "test.somepage.com" + }, + "user": { }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "tmax": 500, + "ext": { + "prebid": { + "aliases": { + "districtm": "appnexus" + }, + "bidadjustmentfactors": { + "appnexus": 1.01, + "districtm": 0.98, + "rubicon": 0.99 + }, + "cache": { + "bids": {} + }, + "targeting": { + "includewinners": false, + "pricegranularity": { + "precision": 2, + "ranges": [ + { + "max": 20, + "increment": 0.10 + } + ] + } + } + } + } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/aliased/multiple-alias.json b/endpoints/openrtb2/sample-requests/aliased/multiple-alias.json new file mode 100644 index 00000000000..55e45041e6e --- /dev/null +++ b/endpoints/openrtb2/sample-requests/aliased/multiple-alias.json @@ -0,0 +1,93 @@ +{ + "description": "Imp extension comes with a valid bidder name and valid bidder aliases as defined in the config.aliases list. Given that 'alias1' refers to the 'appnexus' bidder, we only bid appnexus once.", + "config": { + "aliases": "{\"ext\":{\"prebid\":{\"aliases\":{\"alias1\":\"appnexus\",\"alias2\":\"rubicon\"}}}}" + }, + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + }, + "alias1": { + "placementId": 12883451 + }, + "alias2": { + "accountId": 1001, + "siteId": 113932, + "zoneId": 535510 + } + } + } + ], + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} + } + } + } + }, + "expectedBidResponse": { + "id": "some-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "alias1-bid", + "impid": "", + "price": 0 + } + ], + "seat": "alias1-bids" + }, + { + "bid": [ + { + "id": "alias2-bid", + "impid": "", + "price": 0 + } + ], + "seat": "alias2-bids" + }, + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ], + "bidid": "test bid id", + "nbr": 0 + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/aliased/simple.json b/endpoints/openrtb2/sample-requests/aliased/simple.json index e7f6ba21b83..a99907ab370 100644 --- a/endpoints/openrtb2/sample-requests/aliased/simple.json +++ b/endpoints/openrtb2/sample-requests/aliased/simple.json @@ -1,4 +1,9 @@ { + "description": "Imp extension doesn't come with valid bidder name but does come with valid bidder alias as defined in the mockAliases list.", + "config": { + "aliases": "{\"ext\":{\"prebid\":{\"aliases\":{\"alias1\":\"appnexus\"}}}}" + }, + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -12,10 +17,29 @@ ] }, "ext": { - "test1": { + "alias1": { "placementId": 12883451 } } } ] - } \ No newline at end of file + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0, + "seatbid": [ + { + "bid": [ + { + "id": "alias1-bid", + "impid": "", + "price": 0 + } + ], + "seat": "alias1-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-acct.json b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-acct.json deleted file mode 100644 index ee04a9464e9..00000000000 --- a/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-acct.json +++ /dev/null @@ -1,88 +0,0 @@ -{ - "description": "This is a perfectly valid request except that it comes from a blacklisted Account", - "message": "Invalid request: Prebid-server has blacklisted Account ID: bad_acct, please reach out to the prebid server host.\n", - - "requestPayload": { - "id": "some-request-id", - "user": { - "ext": { - "consent": "gdpr-consent-string", - "prebid": { - "buyeruids": { - "appnexus": "override-appnexus-id-in-cookie" - } - } - } - }, - "app": { - "id": "cool_app", - "publisher": { - "id": "bad_acct" - } - }, - "regs": { - "ext": { - "gdpr": 1 - } - }, - "imp": [ - { - "id": "some-impression-id", - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 600 - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - }, - "districtm": { - "placementId": 105 - }, - "rubicon": { - "accountId": 1001, - "siteId": 113932, - "zoneId": 535510 - } - } - } - ], - "tmax": 500, - "ext": { - "prebid": { - "aliases": { - "districtm": "appnexus" - }, - "bidadjustmentfactors": { - "appnexus": 1.01, - "districtm": 0.98, - "rubicon": 0.99 - }, - "cache": { - "bids": {} - }, - "targeting": { - "includewinners": false, - "pricegranularity": { - "precision": 2, - "ranges": [ - { - "max": 20, - "increment": 0.10 - } - ] - } - } - } - } - } - } - diff --git a/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app-publisher.json b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app-publisher.json new file mode 100644 index 00000000000..ef7a93b8bad --- /dev/null +++ b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app-publisher.json @@ -0,0 +1,90 @@ +{ + "description": "This is a perfectly valid request except that it comes with a blacklisted app publisher account", + "config": { + "blacklistedAccts": ["bad_acct"] + }, + "mockBidRequest": { + "id": "some-request-id", + "user": { + "ext": { + "consent": "gdpr-consent-string", + "prebid": { + "buyeruids": { + "appnexus": "override-appnexus-id-in-cookie" + } + } + } + }, + "app": { + "id": "cool_app", + "publisher": { + "id": "bad_acct" + } + }, + "regs": { + "ext": { + "gdpr": 1 + } + }, + "imp": [ + { + "id": "some-impression-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + }, + "districtm": { + "placementId": 105 + }, + "rubicon": { + "accountId": 1001, + "siteId": 113932, + "zoneId": 535510 + } + } + } + ], + "tmax": 500, + "ext": { + "prebid": { + "aliases": { + "districtm": "appnexus" + }, + "bidadjustmentfactors": { + "appnexus": 1.01, + "districtm": 0.98, + "rubicon": 0.99 + }, + "cache": { + "bids": {} + }, + "targeting": { + "includewinners": false, + "pricegranularity": { + "precision": 2, + "ranges": [ + { + "max": 20, + "increment": 0.10 + } + ] + } + } + } + } + }, + "expectedReturnCode": 503, + "expectedErrorMessage": "Invalid request: Prebid-server has disabled Account ID: bad_acct, please reach out to the prebid server host.\n" +} diff --git a/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app.json b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app.json index 1ace4b53666..120fcec08f4 100644 --- a/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app.json +++ b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-app.json @@ -1,8 +1,9 @@ { "description": "This is a perfectly valid request except that it comes from a blacklisted App", - "message": "Invalid request: Prebid-server does not process requests from App ID: spam_app\n", - - "requestPayload": { + "config": { + "blacklistedApps": ["spam_app"] + }, + "mockBidRequest": { "id": "some-request-id", "user": { "ext": { @@ -80,5 +81,7 @@ } } } - } + }, + "expectedReturnCode": 503, + "expectedErrorMessage": "Invalid request: Prebid-server does not process requests from App ID: spam_app\n" } diff --git a/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-site-publisher.json b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-site-publisher.json new file mode 100644 index 00000000000..cdec20d22ba --- /dev/null +++ b/endpoints/openrtb2/sample-requests/blacklisted/blacklisted-site-publisher.json @@ -0,0 +1,90 @@ +{ + "description": "This is a perfectly valid request except that it comes with a blacklisted site publisher account", + "config": { + "blacklistedAccts": ["bad_acct"] + }, + "mockBidRequest": { + "id": "some-request-id", + "user": { + "ext": { + "consent": "gdpr-consent-string", + "prebid": { + "buyeruids": { + "appnexus": "override-appnexus-id-in-cookie" + } + } + } + }, + "site": { + "id": "cool_site", + "publisher": { + "id": "bad_acct" + } + }, + "regs": { + "ext": { + "gdpr": 1 + } + }, + "imp": [ + { + "id": "some-impression-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 600 + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + }, + "districtm": { + "placementId": 105 + }, + "rubicon": { + "accountId": 1001, + "siteId": 113932, + "zoneId": 535510 + } + } + } + ], + "tmax": 500, + "ext": { + "prebid": { + "aliases": { + "districtm": "appnexus" + }, + "bidadjustmentfactors": { + "appnexus": 1.01, + "districtm": 0.98, + "rubicon": 0.99 + }, + "cache": { + "bids": {} + }, + "targeting": { + "includewinners": false, + "pricegranularity": { + "precision": 2, + "ranges": [ + { + "max": 20, + "increment": 0.10 + } + ] + } + } + } + } + }, + "expectedReturnCode": 503, + "expectedErrorMessage": "Invalid request: Prebid-server has disabled Account ID: bad_acct, please reach out to the prebid server host.\n" +} diff --git a/endpoints/openrtb2/sample-requests/disabled/bad/bad-alias.json b/endpoints/openrtb2/sample-requests/disabled/bad/bad-alias.json index 096c028cfe9..f4379dc09a2 100644 --- a/endpoints/openrtb2/sample-requests/disabled/bad/bad-alias.json +++ b/endpoints/openrtb2/sample-requests/disabled/bad/bad-alias.json @@ -1,6 +1,9 @@ { - "message": "Invalid request: request.ext.prebid.aliases.test1 refers to unknown bidder: appnexus\n", - "requestPayload": { + "description": "Request comes with an alias to a disabled bidder, we should throw error", + "config": { + "disabledAdapters": ["appnexus", "rubicon"] + }, + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -27,5 +30,7 @@ } } } - } -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.ext.prebid.aliases.test1 refers to unknown bidder: appnexus\n" +} diff --git a/endpoints/openrtb2/sample-requests/disabled/bad/bad-bidder.json b/endpoints/openrtb2/sample-requests/disabled/bad/bad-bidder.json index 5f637b1a4fc..91e760ee41e 100644 --- a/endpoints/openrtb2/sample-requests/disabled/bad/bad-bidder.json +++ b/endpoints/openrtb2/sample-requests/disabled/bad/bad-bidder.json @@ -1,6 +1,9 @@ { - "message": "Invalid request: Bidder \"appnexus\" has been disabled on this instance of Prebid Server. Please work with the PBS host to enable this bidder again.\nInvalid request: request.imp[0].ext must contain at least one bidder with valid parameters\n", - "requestPayload": { + "description": "Bid request targeted towards a disabled adapter. We expect an error.", + "config": { + "disabledAdapters": ["appnexus"] + }, + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -20,5 +23,7 @@ } } ] - } -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Bidder \"appnexus\" has been disabled on this instance of Prebid Server. Please work with the PBS host to enable this bidder again.\nInvalid request: request.imp[0].ext must contain at least one bidder\n" +} diff --git a/endpoints/openrtb2/sample-requests/disabled/good/partial.json b/endpoints/openrtb2/sample-requests/disabled/good/partial.json index fe0c492be2d..3549abaa934 100644 --- a/endpoints/openrtb2/sample-requests/disabled/good/partial.json +++ b/endpoints/openrtb2/sample-requests/disabled/good/partial.json @@ -1,4 +1,9 @@ { + "description": "Request comes with some imps directed toward disabled adapters, but there's one non-disabled adapter and we expect a successful response", + "config": { + "disabledAdapters": ["appnexus", "rubicon"] + }, + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -49,4 +54,23 @@ } } } - } \ No newline at end of file + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0, + "seatbid": [ + { + "bid": [ + { + "id": "openx-bid", + "impid": "", + "price": 0 + } + ], + "seat": "openx-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-ext-bidder.json b/endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-ext-bidder.json new file mode 100644 index 00000000000..74dede0857f --- /dev/null +++ b/endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-ext-bidder.json @@ -0,0 +1,50 @@ +{ + "description": "The imp.ext.context field is valid for First Party Data and should be exempted from bidder name validation.", + + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "expectedBidResponse": { + "id":"some-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ], + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-prebid-bidder.json b/endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-prebid-bidder.json new file mode 100644 index 00000000000..41461813c40 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/first-party-data/valid-context-allowed-with-prebid-bidder.json @@ -0,0 +1,54 @@ +{ + "description": "The imp.ext.context field is valid for First Party Data and should be exempted from bidder name validation.", + + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "prebid": { + "bidder": { + "appnexus": { + "placementId": 12883451 + } + } + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "expectedBidResponse": { + "id":"some-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ], + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-data-invalid-type.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-data-invalid-type.json index 06eb73593b4..70bcd4bb94e 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-data-invalid-type.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-data-invalid-type.json @@ -1,11 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "data": { - "type": 364 - } + "description": "Native request with an invalid value in its imp.native.request.data.type field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"data\":{\"type\":364}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-data-no-type.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-data-no-type.json index 6dfe9c400dd..02fcd52be5a 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-data-no-type.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-data-no-type.json @@ -1,9 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "data": {} + "description": "Native request with missing imp.native.request.data.type field value", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"data\":{}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-empty.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-empty.json index 78e2feb7c79..476d1d8fef5 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-empty.json @@ -1,5 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [] -} \ No newline at end of file + "description": "Native request with an empty assets array in its imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-h-negative.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-h-negative.json index 07133c6824b..dec97928777 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-h-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-h-negative.json @@ -1,12 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "img": { - "h": -30, - "w": 20 - } + "description": "Native request with a negative height for its image asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"h\":-30,\"w\":20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-hmin-negative.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-hmin-negative.json index d0654882937..eed0a4905e0 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-hmin-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-hmin-negative.json @@ -1,12 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "img": { - "hmin": -30, - "wmin": 20 - } + "description": "Native request with a negative hmin value in its image asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"hmin\":-30,\"wmin\":20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-w-negative.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-w-negative.json index 8724b76ce24..b7a75f13fe2 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-w-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-w-negative.json @@ -1,12 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "img": { - "h": 30, - "w": -20 - } + "description": "Native request with a negative width value in its image asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"h\":30,\"w\":-20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-wmin-negative.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-wmin-negative.json index 3b81cfd6cb5..ddc8d502287 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-img-wmin-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-img-wmin-negative.json @@ -1,12 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "img": { - "hmin": 30, - "wmin": -20 - } + "description": "Native request with a negative wmin value in its image asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"hmin\":30,\"wmin\":-20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-mixed-type.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-mixed-type.json index 90642cbac18..cfe531af026 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-mixed-type.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-mixed-type.json @@ -1,15 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 140 - }, - "img": { - "wmin": 20, - "hmin": 30 - } + "description": "Native request with mixed asset type in the sole element of the assets arary in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":140},\"img\":{\"wmin\":20,\"hmin\":30}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-title-empty.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-title-empty.json new file mode 100644 index 00000000000..061af88e147 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-title-empty.json @@ -0,0 +1,25 @@ +{ + "description": "Native request with missing length property in its title asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-title-no-length.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-title-no-length.json deleted file mode 100644 index 4b227e3f47f..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-title-no-length.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": {} - } - ] -} \ No newline at end of file diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-mimes-empty.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-mimes-empty.json index 74be7817ceb..649f4c5268b 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-mimes-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-mimes-empty.json @@ -1,14 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "mimes": [], - "minduration": 30, - "maxduration": 120, - "protocols": [3] - } + "description": "Native request with empty mimes array in its video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"mimes\":[],\"minduration\":30,\"maxduration\":120,\"protocols\":[3]}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-maxduration.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-maxduration.json index 5b733aba518..d1faa83b2e4 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-maxduration.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-maxduration.json @@ -1,13 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "mimes": ["video/mp4"], - "minduration": 30, - "protocols": [3] - } + "description": "Native request with missing maxduration value in a video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":30,\"protocols\":[3]}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-mimes.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-mimes.json index 270dccb7cf3..d8792ac7ab3 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-mimes.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-mimes.json @@ -1,13 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "minduration": 30, - "maxduration": 120, - "protocols": [3] - } + "description": "Native request with missing mimes array in a video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"minduration\":30,\"maxduration\":120,\"protocols\":[3]}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-minduration.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-minduration.json index 28f5e7de1c8..d11786b6bc7 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-minduration.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-minduration.json @@ -1,13 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "mimes": ["video/mp4"], - "maxduration": 120, - "protocols": [3] - } + "description": "Native request with missing minduration value in a video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"mimes\":[\"video/mp4\"],\"maxduration\":120,\"protocols\":[3]}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-protocols.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-protocols.json index 59d6f6a5541..adaed92254f 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-protocols.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-no-protocols.json @@ -1,13 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "mimes": ["video/mp4"], - "minduration": 30, - "maxduration": 120 - } + "description": "Native request with missing protocols array in a video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":30,\"maxduration\":120}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-empty.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-empty.json index 4f7616f9da2..018e0168537 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-empty.json @@ -1,14 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "mimes": ["video/mp4"], - "minduration": 30, - "maxduration": 120, - "protocols": [] - } + "description": "Native request with empty protocols array in a video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":30,\"maxduration\":120,\"protocols\":[]}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-invalid.json b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-invalid.json index eb54c644206..1531e45f3e5 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-invalid.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/asset-video-protocols-invalid.json @@ -1,14 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "video": { - "mimes": ["video/mp4"], - "minduration": 30, - "maxduration": 120, - "protocols": [97] - } + "description": "Native request with an out of scope protocol value in a video asset in the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":30,\"maxduration\":120,\"protocols\":[97]}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/assets-with-dup-ids.json b/endpoints/openrtb2/sample-requests/invalid-native/assets-with-dup-ids.json index dfece8cfb0e..91d8c51a317 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/assets-with-dup-ids.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/assets-with-dup-ids.json @@ -1,18 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "id": 1, - "img": { - "wmin": 30 - } + "description": "Native request listing elements with duplicate ids in the assets array the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"id\":1,\"img\":{\"wmin\":30}},{\"id\":1,\"title\":{\"len\":20}}]}" }, - { - "id": 1, - "title": { - "len": 20 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/assets-with-partial-ids.json b/endpoints/openrtb2/sample-requests/invalid-native/assets-with-partial-ids.json index 291ae8d77b1..a485532e042 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/assets-with-partial-ids.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/assets-with-partial-ids.json @@ -1,22 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "id": 1, - "img": { - "wmin": 30 - } + "description": "Native request listing some assets array elements with no id value inside the imp.native.request field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"id\":1,\"img\":{\"wmin\":30}},{\"title\":{\"len\":20}},{\"img\":{\"wmin\":50}}]}" }, - { - "title": { - "len": 20 - } - }, - { - "img": { - "wmin": 50 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-invalid.json b/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-invalid.json index 89a9f83eae0..395e7034b0c 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-invalid.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-invalid.json @@ -1,31 +1,25 @@ { - "context": 1, - "contextsubtype": 21, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } + "description": "Bid request with invalid contextsubtype value inside the native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"contextsubtype\":21,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}]}" }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-negative.json b/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-negative.json index ead42f5701e..c37edd6bb08 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/contextsubtype-negative.json @@ -1,31 +1,25 @@ { - "context": 1, - "contextsubtype": -1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } + "description": "Bid request with a negative contextsubtype value inside the native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"contextsubtype\":-1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}]}" }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/empty-object.json b/endpoints/openrtb2/sample-requests/invalid-native/empty-object.json index 9e26dfeeb6e..3833c25746b 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/empty-object.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/empty-object.json @@ -1 +1,25 @@ -{} \ No newline at end of file +{ + "description": "Bid request with an empty native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].native.request.assets must be an array containing at least one object" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/empty.json b/endpoints/openrtb2/sample-requests/invalid-native/empty.json index e69de29bb2d..4193ebeedd2 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/empty.json @@ -0,0 +1,25 @@ +{ + "description": "Bid request with an empty native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-empty.json b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-empty.json index 8c133fe9eca..d9e10ad2179 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-empty.json @@ -1,31 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } + "description": "Bid request with empty eventtrackers array element inside the native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}],\"eventtrackers\":[{}]}" }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } - ], - "eventtrackers": [{}] + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-event-large.json b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-event-large.json new file mode 100644 index 00000000000..1cdc3c8bbc7 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-event-large.json @@ -0,0 +1,25 @@ +{ + "description": "Bid request with empty eventtrackers array inside the native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}],\"eventtrackers\":[{\"event\":5,\"methods\":[2]}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-empty.json b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-empty.json index f5a48710da7..90c49413ef5 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-empty.json @@ -1,34 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } + "description": "Bid request with empty methods array inside the native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}],\"eventtrackers\":[{\"event\":1,\"methods\":[]}]}" }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } - ], - "eventtrackers": [{ - "event": 1, - "methods": [] - }] + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-large.json b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-large.json index f5a48710da7..8b148b1fa27 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-large.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-methods-large.json @@ -1,34 +1,25 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } + "description": "Bid request with empty methods array inside the native.request field in its only imp element", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}],\"eventtrackers\":[{\"event\":1,\"methods\":[3]}]}" }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } - ], - "eventtrackers": [{ - "event": 1, - "methods": [] - }] + } + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" } diff --git a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-type-large.json b/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-type-large.json deleted file mode 100644 index d0d666ac186..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-native/eventtracker-type-large.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } - }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } - } - ], - "eventtrackers": [{ - "event": 1, - "methods": [5] - }] -} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/request-context-invalid.json b/endpoints/openrtb2/sample-requests/invalid-native/request-context-invalid.json index e005815ad3e..09da5d165ae 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/request-context-invalid.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/request-context-invalid.json @@ -1,12 +1,25 @@ { - "context": 376, - "plcmttype": 2, - "assets": [ - { - "img": { - "hmin": 30, - "wmin": 20 - } + "description": "Native request with an imp.native.request.context value that doesn't match that of the context_sub_type list", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":376,\"plcmttype\":2,\"assets\":[{\"img\":{\"hmin\":30,\"wmin\":20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-native/request-plcmttype-invalid.json b/endpoints/openrtb2/sample-requests/invalid-native/request-plcmttype-invalid.json index e34b17e5d4d..c2104bcd4b5 100644 --- a/endpoints/openrtb2/sample-requests/invalid-native/request-plcmttype-invalid.json +++ b/endpoints/openrtb2/sample-requests/invalid-native/request-plcmttype-invalid.json @@ -1,12 +1,25 @@ { - "context": 1, - "plcmttype": 423, - "assets": [ - { - "img": { - "hmin": 30, - "wmin": 20 - } + "description": "Native request with an imp.native.request.plcmttype value that doesn't match that of the placement_type list", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":423,\"assets\":[{\"img\":{\"hmin\":30,\"wmin\":20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_1.json b/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_1.json index 2a647d7d8c8..812f0664fbb 100644 --- a/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_1.json +++ b/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_1.json @@ -1,41 +1,15 @@ { - "description": "Otherwise valid request using stored request; incoming request has a comma after the closing curly brace of the second set of dimensions to yield invalid JSON", - - "message": "Invalid request: Invalid JSON in Incoming Request: invalid character ']' looking for beginning of value at offset 377\n", - "requestPayload": { - "id": "some-request-id", - "site": { - "page": "prebid.org" - }, - "imp": [ - { - "id": "some-impression-id", - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 600 - }, - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "tmax": 500, - "ext": { - "prebid": { - "storedrequest": { - "id": "2" - } + "description": "Otherwise valid request using stored request; incoming request has a comma after the closing curly brace of the second set of dimensions to yield invalid JSON", + "mockBidRequest": { + "imp": [{},], + "ext": { + "prebid": { + "storedrequest": { + "id": "2" } } } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Invalid JSON in Incoming Request: invalid character ']' looking for beginning of value at offset" } diff --git a/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_2.json b/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_2.json deleted file mode 100644 index d18a20d7a13..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_2.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "description": "Otherwise valid request using stored request; incoming request lacks a comma after first width to yield invalid JSON", - - "message": "Invalid request: Invalid JSON in Incoming Request: invalid character '\"' after object key:value pair at offset 254\n", - "requestPayload": { - "id": "some-request-id", - "site": { - "page": "prebid.org" - }, - "imp": [ - { - "id": "some-impression-id", - "banner": { - "format": [ - { - "w": 300 - "h": 250 - }, - { - "w": 300, - "h": 600 - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "tmax": 500, - "ext": { - "prebid": { - "storedrequest": { - "id": "2" - } - } - } - } -} diff --git a/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_imp.json b/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_imp.json index 0898b9da55a..e3960b17399 100644 --- a/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_imp.json +++ b/endpoints/openrtb2/sample-requests/invalid-stored/bad_incoming_imp.json @@ -1,8 +1,7 @@ { "description": "Otherwise valid request but without comma after first width field in first imp to yield invalid JSON, using stored imp request", - "message": "Invalid request: Invalid JSON in Imp[0] of Incoming Request: invalid character '\"' after object key:value pair at offset 132\n", - "requestPayload": { + "mockBidRequest": { "id": "some-request-id", "site": { "page": "prebid.org" @@ -36,5 +35,7 @@ } ], "tmax": 500 - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Invalid JSON in Imp[0] of Incoming Request: invalid character '\"' after object key:value pair at offset 132\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_imp.json b/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_imp.json index 1847fd4108a..0fed6c32adf 100644 --- a/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_imp.json +++ b/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_imp.json @@ -1,8 +1,7 @@ { "description": "Valid request using stored imp request which has missing comma to yield invalid JSON", - "message": "Invalid request: imp.ext.prebid.storedrequest.id 7: Stored Imp has Invalid JSON: invalid character '\"' after object key:value pair at offset 185\n", - "requestPayload": { + "mockBidRequest": { "id": "some-request-id", "site": { "page": "prebid.org" @@ -32,5 +31,7 @@ } ], "tmax": 500 - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: imp.ext.prebid.storedrequest.id 7: Stored Imp has Invalid JSON" } diff --git a/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_req.json b/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_req.json index 46df45d67da..4e9d7f03352 100644 --- a/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_req.json +++ b/endpoints/openrtb2/sample-requests/invalid-stored/bad_stored_req.json @@ -1,26 +1,27 @@ { - "description": "Valid request that uses stored request with extra curly brace so stored request is not valid JSON", + "description": "Valid request that uses stored request with extra curly brace so stored request is not valid JSON", - "message": "Invalid request: ext.prebid.storedrequest.id refers to Stored Request 3 which contains Invalid JSON: invalid character '}' after top-level value at offset 293\n", - "requestPayload": { - "id": "ThisID", - "imp": [ - { - "ext": { - "prebid": { - "storedrequest": { - "id": "1" - } + "mockBidRequest": { + "id": "ThisID", + "imp": [ + { + "ext": { + "prebid": { + "storedrequest": { + "id": "1" } } } - ], - "ext": { - "prebid": { - "storedrequest": { - "id": "3" - } + } + ], + "ext": { + "prebid": { + "storedrequest": { + "id": "3" } } } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: ext.prebid.storedrequest.id refers to Stored Request 3 which contains Invalid JSON" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/alias-bidder-self.json b/endpoints/openrtb2/sample-requests/invalid-whole/alias-bidder-self.json index 666253ec85b..5e0aa5a989b 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/alias-bidder-self.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/alias-bidder-self.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.ext.prebid.aliases.appnexus defines a no-op alias. Choose a different alias, or remove this entry.\n", - "requestPayload": { + "description": "Request with invalid alias in the root extension field", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -20,10 +20,12 @@ ], "ext": { "prebid": { - "aliases": { - "appnexus": "appnexus" - } + "aliases": { + "appnexus": "appnexus" + } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.ext.prebid.aliases.appnexus defines a no-op alias. Choose a different alias, or remove this entry.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/alias-unknown-core.json b/endpoints/openrtb2/sample-requests/invalid-whole/alias-unknown-core.json index 6c1925d65b1..f7b5597d59f 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/alias-unknown-core.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/alias-unknown-core.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.ext.prebid.aliases.unknown refers to unknown bidder: other-unknown\n", - "requestPayload": { + "description": "Request targets an unknown bidder", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -11,7 +11,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "unknown": { @@ -27,5 +27,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.ext.prebid.aliases.unknown refers to unknown bidder: other-unknown\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/app-bad-ext.json b/endpoints/openrtb2/sample-requests/invalid-whole/app-bad-ext.json deleted file mode 100644 index 672b05724c8..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/app-bad-ext.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "message": "Invalid request: json: cannot unmarshal object into Go struct field ExtAppPrebid.source of type string\n", - "requestPayload": { - "id": "some-request-id", - "app": { - "ext": { - "prebid": { - "source": { - "foo": "prebid-mobile" - }, - "version": "1.0" - } - } - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ] - } -} \ No newline at end of file diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/array.json b/endpoints/openrtb2/sample-requests/invalid-whole/array.json deleted file mode 100644 index bbdf8c57de8..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/array.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "message": "Invalid request: json: cannot unmarshal array into Go value of type openrtb.BidRequest\n", - "requestPayload": [] -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/audio-mimes-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/audio-mimes-empty.json index 2e0e299923a..6ade27dd926 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/audio-mimes-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/audio-mimes-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].audio.mimes must contain at least one supported MIME type\n", - "requestPayload": { + "description": "Empty request.imp[0].audio.mimes array", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -13,5 +13,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].audio.mimes must contain at least one supported MIME type\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-only.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-only.json index 515824e65d9..00e7bbbbc68 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-only.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-only.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n", - "requestPayload": { + "description": "Request invalid because banner is missing width value", + "mockBidRequest": { "id":"req-id", "site": { "id": "some-site" @@ -18,5 +18,7 @@ } } ] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-zero.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-zero.json deleted file mode 100644 index f18f63e5e28..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-h-zero.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "message": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n", - "requestPayload": { - "id":"req-id", - "site": { - "id": "some-site" - }, - "imp": [ - { - "id":"imp-id", - "banner":{ - "h": 0, - "w": 250 - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ] - } -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmax.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmax.json index 1cb04fc5c1d..d97f358c328 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmax.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmax.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].banner uses unsupported property: \"hmax\". Use the \"format\" array instead.\n", - "requestPayload": { + "description": "Request comes with unsupported property 'hmax'", + "mockBidRequest": { "id":"req-id", "imp": [ { @@ -13,5 +13,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner uses unsupported property: \"hmax\". Use the \"format\" array instead.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmin.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmin.json index 38eeeb612e0..5e65b845bc9 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmin.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/banner-hmin.json @@ -1,17 +1,19 @@ { - "message": "Invalid request: request.imp[0].banner uses unsupported property: \"hmin\". Use the \"format\" array instead.\n", - "requestPayload": { - "id":"req-id", - "imp": [ - { - "id":"imp-id", - "banner": { - "hmin":50 - } + "description": "Request comes with unsupported property 'hmin'", + "mockBidRequest": { + "id":"req-id", + "imp": [ + { + "id":"imp-id", + "banner": { + "hmin":50 } - ], - "app": { - "id": "app_001" } + ], + "app": { + "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner uses unsupported property: \"hmin\". Use the \"format\" array instead.\n" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-null.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-null.json deleted file mode 100644 index bd8beef8868..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-null.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "message": "Invalid request: request.imp[0] must contain at least one of \"banner\", \"video\", \"audio\", or \"native\"\n", - "requestPayload": { - "id":"req-id", - "imp": [ - { - "id": "imp-id", - "banner": null - } - ], - "app": { - "id": "app_001" - } - } -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-only.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-only.json index 70739a65834..7f1abcf7de8 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-only.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-only.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n", - "requestPayload": { + "description": "Request comes with a banner that defines no 'h' value", + "mockBidRequest": { "id":"req-id", "site": { "id": "some-site" @@ -18,5 +18,7 @@ } } ] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-zero.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-zero.json deleted file mode 100644 index b3453ab4cb7..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-w-zero.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "message": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n", - "requestPayload": { - "id":"req-id", - "site": { - "id": "some-site" - }, - "imp": [ - { - "id": "imp-id", - "banner": { - "h": 300, - "w": 0 - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ] - } -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmax.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmax.json index f1b7d0aeb96..3d30926ce34 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmax.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmax.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].banner uses unsupported property: \"wmax\". Use the \"format\" array instead.\n", - "requestPayload": { + "description": "Request comes with unsupported property 'wmax'", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -13,5 +13,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner uses unsupported property: \"wmax\". Use the \"format\" array instead.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmin.json b/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmin.json index a39e1539ab9..235ff94a031 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmin.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/banner-wmin.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].banner uses unsupported property: \"wmin\". Use the \"format\" array instead.\n", - "requestPayload": { + "description": "Request comes with unsupported property 'wmin'", + "mockBidRequest": { "id":"req-id", "imp": [ { @@ -13,5 +13,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner uses unsupported property: \"wmin\". Use the \"format\" array instead.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-invalid-bidder.json b/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-invalid-bidder.json index 569e16d2d20..90580bf65ff 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-invalid-bidder.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-invalid-bidder.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.ext.prebid.bidadjustmentfactors.unknown is not a known bidder or alias\n", - "requestPayload": { + "description": "Bid adjustment factor assigned to an unknown bidder", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -25,5 +25,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.ext.prebid.bidadjustmentfactors.unknown is not a known bidder or alias\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-negative.json b/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-negative.json index 4db6ee09bd8..43d51ac20cb 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/bid-adjustment-negative.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.ext.prebid.bidadjustmentfactors.appnexus must be a positive number. Got -2.000000\n", - "requestPayload": { + "description": "Negative bid adjustment factor", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -25,5 +25,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.ext.prebid.bidadjustmentfactors.appnexus must be a positive number. Got -2.000000\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/boolean.json b/endpoints/openrtb2/sample-requests/invalid-whole/boolean.json deleted file mode 100644 index a807200f732..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/boolean.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "message": "Invalid request: json: cannot unmarshal bool into Go value of type openrtb.BidRequest\n", - "requestPayload": false -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/cache-nothing.json b/endpoints/openrtb2/sample-requests/invalid-whole/cache-nothing.json index d4b875498ae..76dc1c2bb5c 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/cache-nothing.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/cache-nothing.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.ext is invalid: request.ext.prebid.cache requires one of the \"bids\" or \"vastml\" properties\n", - "requestPayload": { + "description": "Empty cache field in root level extension", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -11,7 +11,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "appnexus": { @@ -25,5 +25,7 @@ "cache": {} } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.ext is invalid: request.ext.prebid.cache requires one of the \"bids\" or \"vastxml\" properties\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/deal-no-id.json b/endpoints/openrtb2/sample-requests/invalid-whole/deal-no-id.json index ffb3e19cfbc..5c992b49d38 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/deal-no-id.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/deal-no-id.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].pmp.deals[0] missing required field: \"id\"\n", - "requestPayload": { + "description": "Empty id field in the deals array of the pmp field", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -8,7 +8,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "pmp": { "deals": [ @@ -23,5 +23,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].pmp.deals[0] missing required field: \"id\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/digitrust.json b/endpoints/openrtb2/sample-requests/invalid-whole/digitrust.json index 1be93853a0b..1fb7169fced 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/digitrust.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/digitrust.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user contains a digitrust object that is not valid.\n", - "requestPayload": { + "description": "Invalid digitrust object in user extension", + "mockBidRequest": { "id": "request-with-invalid-digitrust-obj", "site": { "page": "test.somepage.com" @@ -40,5 +40,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user contains a digitrust object that is not valid.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/empty-object.json b/endpoints/openrtb2/sample-requests/invalid-whole/empty-object.json index a17e6d160e5..26c54e6828d 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/empty-object.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/empty-object.json @@ -1,4 +1,6 @@ { - "message": "Invalid request: request missing required field: \"id\"\n", - "requestPayload": {} + "message": "Empty bid request, expect error", + "mockBidRequest": {}, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request missing required field: \"id\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/float.json b/endpoints/openrtb2/sample-requests/invalid-whole/float.json deleted file mode 100644 index 489f258a0f2..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/float.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "message": "Invalid request: json: cannot unmarshal number into Go value of type openrtb.BidRequest\n", - "requestPayload": 6.3 -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-array.json b/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-array.json index 15e41cc5fb2..94722cc3253 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-array.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-array.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n", - "requestPayload": { + "description": "Banner with empty format array does not define width nor height in w and h fields", + "mockBidRequest": { "id":"req-id", "site": { "id": "some-site" @@ -18,5 +18,7 @@ } } ] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].banner has no sizes. Define \"w\" and \"h\", or include \"format\" elements.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-object.json b/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-object.json index 9117cad5d45..369722f453d 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-object.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/format-empty-object.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: Request imp[0].banner.format[0] should define *either* {w, h} (for static size requirements) *or* {wmin, wratio, hratio} (for flexible sizes) to be non-zero.\n", - "requestPayload": { + "description": "Banner does not define width nor height in format array element nor comes with w and h field values", + "mockBidRequest": { "id":"req-id", "imp": [ { @@ -13,5 +13,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Request imp[0].banner.format[0] should define *either* {w, h} (for static size requirements) *or* {wmin, wratio, hratio} (for flexible sizes) to be non-zero.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/format-no-height.json b/endpoints/openrtb2/sample-requests/invalid-whole/format-no-height.json index 5b5b2e64e29..919504d51ee 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/format-no-height.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/format-no-height.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: Request imp[0].banner.format[0] must define non-zero \"h\" and \"w\" properties.\n", - "requestPayload": { + "description": "Banner comes with a zero height value in format array element", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -18,5 +18,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Request imp[0].banner.format[0] must define non-zero \"h\" and \"w\" properties.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/format-no-hratio.json b/endpoints/openrtb2/sample-requests/invalid-whole/format-no-hratio.json index d5a404d2059..49756a5b708 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/format-no-hratio.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/format-no-hratio.json @@ -1,21 +1,23 @@ { - "message": "Invalid request: Request imp[0].banner.format[0] must define non-zero \"wmin\", \"wratio\", and \"hratio\" properties.\n", - "requestPayload": { - "id": "req-id", - "imp": [ - { - "id": "imp-id", - "banner": { - "format": [ - { - "wratio": 30 - } - ] - } - } - ], - "app": { - "id": "app_001" + "description": "Banner comes with a zero hratio value", + "mockBidRequest": { + "id": "req-id", + "imp": [ + { + "id": "imp-id", + "banner": { + "format": [ + { + "wratio": 30 + } + ] } - } + } + ], + "app": { + "id": "app_001" + } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Request imp[0].banner.format[0] must define non-zero \"wmin\", \"wratio\", and \"hratio\" properties.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/format-two-widths.json b/endpoints/openrtb2/sample-requests/invalid-whole/format-two-widths.json index cbe7b4af663..387299b2115 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/format-two-widths.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/format-two-widths.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: Request imp[0].banner.format[0] should define *either* {w, h} *or* {wmin, wratio, hratio}, but not both. If both are valid, send two \"format\" objects in the request.\n", - "requestPayload": { + "description": "Banner specifies both w and wratio values, which is invalid", + "mockBidRequest": { "id":"req-id", "imp": [ { @@ -18,5 +18,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Request imp[0].banner.format[0] should define *either* {w, h} *or* {wmin, wratio, hratio}, but not both. If both are valid, send two \"format\" objects in the request.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-array.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-array.json index f3e3914db3b..9de06382bc8 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-array.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-array.json @@ -1,7 +1,9 @@ { - "message": "Invalid request: request.imp must contain at least one element.\n", - "requestPayload": { + "description": "Bid request comes with an empty Imp array", + "mockBidRequest": { "id": "req-id", "imp": [] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp must contain at least one element.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-object.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-object.json index f2fc508910d..d66d7ed0fae 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-object.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-empty-object.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0] missing required field: \"id\"\n", - "requestPayload": { + "description": "Bid request's sole imp element is empty", + "mockBidRequest": { "id": "req-id", "imp": [ { } @@ -8,5 +8,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0] missing required field: \"id\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-empty.json index 4d199e48b60..6c60ed5def2 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].ext must contain at least one bidder with valid parameters\n", - "requestPayload": { + "description": "Bid request's sole imp element has empty extension", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -8,7 +8,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": {} } @@ -16,5 +16,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].ext must contain at least one bidder\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-invalid-params.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-invalid-params.json index c65e9b3ba59..d9ed59c8c6e 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-invalid-params.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-invalid-params.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].ext.appnexus failed validation.\n(root): Invalid type. Expected: object, given: string\n", - "requestPayload": { + "description": "Bid request's sole imp element has invalid ext value", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -8,7 +8,7 @@ "audio": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "appnexus": "invalidParams" @@ -18,5 +18,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].ext.appnexus failed validation.\n(root): Invalid type. Expected: object, given: string\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-unknown-bidder.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-unknown-bidder.json index 436e62f7174..9f5a45c861d 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-unknown-bidder.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-ext-unknown-bidder.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].ext contains unknown bidder: noBidderShouldEverHaveThisName. Did you forget an alias in request.ext.prebid.aliases?\n", - "requestPayload": { + "description": "Unknown biddername in bid request's sole imp element's ext", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -8,7 +8,7 @@ "audio": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "noBidderShouldEverHaveThisName": { @@ -20,5 +20,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].ext contains unknown bidder: noBidderShouldEverHaveThisName. Did you forget an alias in request.ext.prebid.aliases?\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-id-duplicates.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-id-duplicates.json index 53517c268b6..90e605362fa 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-id-duplicates.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-id-duplicates.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].id and request.imp[1].id are both \"some-impression-id\". Imp IDs must be unique.\n", - "requestPayload": { + "description": "Identical id's in more than one imp element", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "prebid.org" @@ -40,5 +40,7 @@ } ], "tmax": 500 - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].id and request.imp[1].id are both \"some-impression-id\". Imp IDs must be unique.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-ext.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-ext.json index 3249f077c2c..ce2aa197d3e 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-ext.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-ext.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].ext is required\n", - "requestPayload": { + "description": "Bid request's sole imp element has no ext field", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -8,12 +8,14 @@ "video": { "mimes": [ "video/mp4" - ] + ] } } ], "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].ext is required\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-type.json b/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-type.json index c7b005ca5d3..713e009b51d 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-type.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/imp-no-type.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0] must contain at least one of \"banner\", \"video\", \"audio\", or \"native\"\n", - "requestPayload": { + "description": "Bid request's sole imp element comes with no Banner, Video, Audio, nor Native field", + "mockBidRequest": { "id":"req-id", "imp": [ { @@ -10,5 +10,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0] must contain at least one of \"banner\", \"video\", \"audio\", or \"native\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/integer.json b/endpoints/openrtb2/sample-requests/invalid-whole/integer.json deleted file mode 100644 index 5eefb89b2a7..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/integer.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "message": "Invalid request: json: cannot unmarshal number into Go value of type openrtb.BidRequest\n", - "requestPayload": 5 -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/interstital-bad-perc.json b/endpoints/openrtb2/sample-requests/invalid-whole/interstital-bad-perc.json index 6854ea9a470..302372b5e5d 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/interstital-bad-perc.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/interstital-bad-perc.json @@ -1,51 +1,53 @@ { - "message": "Invalid request: request.device.ext.prebid.interstitial.minwidthperc must be a number between 0 and 100\n", - "requestPayload": { - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "id": "some-imp-id", - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "instl": 1, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "device": { - "h": 640, - "w": 320, - "ext": { - "prebid": { - "interstitial": { - "minwidthperc": 120, - "minheightperc": 60 - } - } + "description": "Bid request's device field comes with a minwidthperc value greater than 100", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "id": "some-imp-id", + "format": [ + { + "w": 300, + "h": 600 } + ] }, + "instl": 1, "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "device": { + "h": 640, + "w": 320, + "ext": { "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} - } + "interstitial": { + "minwidthperc": 120, + "minheightperc": 60 + } } + } + }, + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} } + } } -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.device.ext.prebid.interstitial.minwidthperc must be a number between 0 and 100\n" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/interstitial-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/interstitial-empty.json index a69f287dfab..8753d50fb99 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/interstitial-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/interstitial-empty.json @@ -1,43 +1,45 @@ { - "message": "Invalid request: Unable to set interstitial size list for Imp id=my-imp-id (No valid sizes between 0x0 and 0x0)\n", - "requestPayload": { - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "id": "some-imp-id" - }, - "instl": 1, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "device": { - "ext": { - "prebid": { - "interstitial": { - "minwidthperc": 60, - "minheightperc": 60 - } - } - } + "description": "Bid request banner field comes with no data to set interstitial size list", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "id": "some-imp-id" }, + "instl": 1, "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "device": { + "ext": { "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} - } + "interstitial": { + "minwidthperc": 60, + "minheightperc": 60 + } } + } + }, + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} } + } } -} \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: Unable to set interstitial size list for Imp id=my-imp-id (No valid sizes between 0x0 and 0x0)\n" +} diff --git a/endpoints/openrtb2/sample-requests/aliased/site.json b/endpoints/openrtb2/sample-requests/invalid-whole/invalid-source.json similarity index 53% rename from endpoints/openrtb2/sample-requests/aliased/site.json rename to endpoints/openrtb2/sample-requests/invalid-whole/invalid-source.json index cf7e9a77533..8385f924a56 100644 --- a/endpoints/openrtb2/sample-requests/aliased/site.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/invalid-source.json @@ -1,7 +1,16 @@ { + "description": "Request with wrong source value", + "mockBidRequest": { "id": "some-request-id", - "site": { - "page": "test.somepage.com" + "app": { + "ext": { + "prebid": { + "source": { + "foo": "prebid-mobile" + }, + "version": "1.0" + } + } }, "imp": [ { @@ -24,27 +33,11 @@ "ext": { "appnexus": { "placementId": 12883451 - }, - "test1": { - "placementId": 12883451 - }, - "test2": { - "accountId": 1001, - "siteId": 113932, - "zoneId": 535510 - } - } - } - ], - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} + } } } - } - } - \ No newline at end of file + ] + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: json: cannot unmarshal object into Go struct field ExtAppPrebid.prebid.source of type string" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/malformed-bid-request.json b/endpoints/openrtb2/sample-requests/invalid-whole/malformed-bid-request.json new file mode 100644 index 00000000000..c6fd52304a7 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/invalid-whole/malformed-bid-request.json @@ -0,0 +1,6 @@ +{ + "description": "Malformed bid request throws an error", + "mockBidRequest": "malformed", + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: invalid character 'm' looking for beginning of value\n" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/metric-empty-object.json b/endpoints/openrtb2/sample-requests/invalid-whole/metric-empty-object.json index b8cc1f7983a..022326a5b02 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/metric-empty-object.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/metric-empty-object.json @@ -1,15 +1,17 @@ { - "message": "Invalid request: request.imp[0].metric is not yet supported by prebid-server. Support may be added in the future\n", - "requestPayload": { - "id":"req-id", - "imp":[ - { - "id":"imp-id", - "metric": [{}] + "description": "Bid request's imp comes with elements in unsupported metric array", + "mockBidRequest": { + "id":"req-id", + "imp":[ + { + "id":"imp-id", + "metric": [{}] + } + ], + "app": { + "id": "app_001" } - ], - "app": { - "id": "app_001" - } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].metric is not yet supported by prebid-server. Support may be added in the future\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/native-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/native-empty.json index fa2bdded0e7..185ad8fd8a5 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/native-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/native-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].native missing required property \"request\"\n", - "requestPayload": { + "description": "Bid request's sole imp element has empty Native field and does not define a Banner, Video nor Audio field", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -11,5 +11,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].native missing required property \"request\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/no-site-or-app.json b/endpoints/openrtb2/sample-requests/invalid-whole/no-site-or-app.json index c56dae324fc..d21ca4a94ae 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/no-site-or-app.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/no-site-or-app.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.site or request.app must be defined, but not both.\n", - "requestPayload": { + "description": "Request does not come with site field nor app field", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -8,7 +8,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "appnexus": { @@ -17,5 +17,7 @@ } } ] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.site or request.app must be defined, but not both.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/null.json b/endpoints/openrtb2/sample-requests/invalid-whole/null.json index eb221dddfeb..ac94de741c8 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/null.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/null.json @@ -1,4 +1,6 @@ { - "message": "Invalid request: request missing required field: \"id\"\n", - "requestPayload": null + "description": "Bid request is null", + "mockBidRequest": null, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request missing required field: \"id\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/only-request-id.json b/endpoints/openrtb2/sample-requests/invalid-whole/only-request-id.json index 9148e1a4d5b..5938cfc0243 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/only-request-id.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/only-request-id.json @@ -1,6 +1,8 @@ { - "message": "Invalid request: request.imp must contain at least one element.\n", - "requestPayload": { + "description": "Bid request is is missing imp array", + "mockBidRequest": { "id": "req-id" - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp must contain at least one element.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-invalid.json b/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-invalid.json index dff3023c702..1847dc72283 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-invalid.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-invalid.json @@ -1,11 +1,11 @@ { - "message": "Invalid request: request.regs.ext.gdpr must be either 0 or 1.\n", - "requestPayload": { + "description": "Bid request defines an invalid GDPR value", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", "publisher": { - "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" + "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" } }, "source": { @@ -23,7 +23,7 @@ "banner": { "format": [ { - "w": 300, + "w": 300, "h": 250 }, { @@ -36,11 +36,13 @@ ], "regs": { "ext": { - "gdpr": 2 + "gdpr": 2 } }, "user": { "ext": {} } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.regs.ext.gdpr must be either 0 or 1.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-string.json b/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-string.json index ce887889034..afdabdab7cf 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-string.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-gdpr-string.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.regs.ext is invalid: json: cannot unmarshal string into Go struct field ExtRegs.gdpr of type int8\n", - "requestPayload": { + "description": "Invalid GDPR value in regs field", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -42,5 +42,7 @@ "user": { "ext": {} } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.regs.ext is invalid: json: cannot unmarshal string into Go struct field ExtRegs.gdpr of type int8\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-malformed.json b/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-malformed.json index a403103d6fb..a8e94008cf1 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-malformed.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/regs-ext-malformed.json @@ -1,45 +1,46 @@ { - "message": "Invalid request: request.regs.ext is invalid: json: cannot unmarshal string into Go value of type openrtb_ext.ExtRegs\n", - "requestPayload": { - "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", - "site": { - "page": "prebid.org", - "publisher": { + "description": "Malformed ext in regs field", + "mockBidRequest": { + "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", + "site": { + "page": "prebid.org", + "publisher": { "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" - } - }, - "source": { - "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" - }, - "tmax": 1000, - "imp": [ - { - "id": "/19968336/header-bid-tag-0", - "ext": { - "appnexus": { - "placementId": 12883451 - } - }, - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 300 - } - ] + } + }, + "source": { + "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" + }, + "tmax": 1000, + "imp": [ + { + "id": "/19968336/header-bid-tag-0", + "ext": { + "appnexus": { + "placementId": 12883451 } + }, + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 300 + } + ] } - ], - "regs": { - "ext": "malformed" - }, - "user": { - "ext": {} } + ], + "regs": { + "ext": "malformed" + }, + "user": { + "ext": {} } - } - \ No newline at end of file + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.regs.ext is invalid: json: cannot unmarshal string into Go value of type openrtb_ext.ExtRegs\n" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/site-app-both.json b/endpoints/openrtb2/sample-requests/invalid-whole/site-app-both.json index 4b643705640..5bc3054c356 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/site-app-both.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/site-app-both.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.site or request.app must be defined, but not both.\n", - "requestPayload": { + "description": "Bid request comes with both site and app fields, it should only come with one or the other", + "mockBidRequest": { "id": "req-id", "site": { "page": "test.mysite.com" @@ -12,7 +12,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "appnexus": { @@ -21,5 +21,7 @@ } } ] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.site or request.app must be defined, but not both.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/site-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/site-empty.json index 3d53314dbb7..80f88abbf04 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/site-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/site-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.site should include at least one of request.site.id or request.site.page.\n", - "requestPayload": { + "description": "Bid request's site field is missing an id", + "mockBidRequest": { "id": "req-id", "site": {}, "imp": [ @@ -9,7 +9,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "appnexus": { @@ -18,5 +18,7 @@ } } ] - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.site should include at least one of request.site.id or request.site.page.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/site-ext-amp.json b/endpoints/openrtb2/sample-requests/invalid-whole/site-ext-amp.json index bebe4625578..6575bb16fe9 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/site-ext-amp.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/site-ext-amp.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.site.ext.amp must be either 1, 0, or undefined\n", - "requestPayload": { + "description": "Request's amp value in site's ext is invalid", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com", @@ -43,5 +43,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.site.ext.amp must be either 1, 0, or undefined\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/storedrequest-id-int.json b/endpoints/openrtb2/sample-requests/invalid-whole/storedrequest-id-int.json index 5d510d21dbd..86e4b9cac5f 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/storedrequest-id-int.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/storedrequest-id-int.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: ext.prebid.storedrequest.id must be a string\n", - "requestPayload": { + "description": "Bid request with an invalid value for ext.prebid.storedrequest.id field", + "mockBidRequest": { "id": "some-request-id", "site": { "page": "test.somepage.com" @@ -11,7 +11,7 @@ "video": { "mimes": [ "video/mp4" - ] + ] }, "ext": { "appnexus": { @@ -27,5 +27,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: ext.prebid.storedrequest.id must be a string\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/tmax-negative.json b/endpoints/openrtb2/sample-requests/invalid-whole/tmax-negative.json index e97a20816f2..8fa52f24ca9 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/tmax-negative.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/tmax-negative.json @@ -1,7 +1,9 @@ { - "message": "Invalid request: request.tmax must be nonnegative. Got -2\n", - "requestPayload": { + "description": "Bid request with negative tmax value. Expect error", + "mockBidRequest": { "id": "req-id", "tmax": -2 - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.tmax must be nonnegative. Got -2\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/unknown-bidder.json b/endpoints/openrtb2/sample-requests/invalid-whole/unknown-bidder.json index 3914ae7ae49..a907d4d9257 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/unknown-bidder.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/unknown-bidder.json @@ -1,38 +1,38 @@ { - "description": "Copy of the prebid test ad, with the addition of an unknown bidder", - - "message": "Invalid request: request.imp[0].ext contains unknown bidder: unknownbidder. Did you forget an alias in request.ext.prebid.aliases?\n", - "requestPayload": { - "id": "some-request-id", - "site": { - "page": "prebid.org" - }, - "imp": [ - { - "id": "some-impression-id", - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 600 - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 + "description": "Copy of the prebid test ad, with the addition of an unknown bidder", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "prebid.org" + }, + "imp": [ + { + "id": "some-impression-id", + "banner": { + "format": [ + { + "w": 300, + "h": 250 }, - "unknownbidder": { - "param1": "foobar", - "param2": 42 + { + "w": 300, + "h": 600 } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + }, + "unknownbidder": { + "param1": "foobar", + "param2": 42 } } - ], - "tmax": 500 - } - } \ No newline at end of file + } + ], + "tmax": 500 + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request" +} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-consent-int.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-consent-int.json index 5bc0ed33eab..b61be105df0 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-consent-int.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-consent-int.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext object is not valid: json: cannot unmarshal number into Go struct field ExtUser.consent of type string\n", - "requestPayload": { + "description": "Bid request comes with an integer value for user.ext.consent instead of a JSON object", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -22,14 +22,14 @@ }, "banner": { "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 300 - } + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 300 + } ] } } @@ -44,5 +44,7 @@ "consent": 1 } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext object is not valid: json: cannot unmarshal number into Go struct field ExtUser.consent of type string\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-eids-uids-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-eids-uids-empty.json index ebbb4e2701c..d1cd1eb512e 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-eids-uids-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-eids-uids-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.eids[0].uids must contain at least one element or be undefined\n", - "requestPayload": { + "description": "Bid request with empty uids array in user.ext.eids array element", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -49,5 +49,7 @@ ] } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.eids[0].uids must contain at least one element or be undefined\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-empty.json index 3d73d73117e..98297739105 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.eids must contain at least one element or be undefined\n", - "requestPayload": { + "description": "Bid request with empty eids array in user.ext", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -44,5 +44,7 @@ "eids": [] } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.eids must contain at least one element or be undefined\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-id-uids-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-id-uids-empty.json index bbd0dadfd70..09ad4eeb68a 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-id-uids-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-id-uids-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.eids[0] must contain either \"id\" or \"uids\" field\n", - "requestPayload": { + "description": "Bid request with user.ext.eids array element array element that does not contain an id nor uids", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -48,5 +48,7 @@ ] } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.eids[0] must contain either \"id\" or \"uids\" field\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-empty.json index 5efff0626ef..06684b5e62e 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.eids[0] missing required field: \"source\"\n", - "requestPayload": { + "description": "Bid request with user.ext.eids array element array element that does not contain source field", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -46,5 +46,7 @@ ] } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.eids[0] missing required field: \"source\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-unique.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-unique.json index e508b113aff..5a6548c3eb2 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-unique.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-source-unique.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.eids must contain unique sources\n", - "requestPayload": { + "description": "Bid request where more than one request.user.ext.eids array elements share the same source field value", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -53,5 +53,7 @@ ] } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.eids must contain unique sources\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-uids-id-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-uids-id-empty.json index 3a9659b7327..faa09f93ad1 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-uids-id-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-uids-id-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.eids[0].uids[0] missing required field: \"id\"\n", - "requestPayload": { + "description": "Bid request where a request.user.ext.eids.uids array element is missing its id field", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -51,5 +51,7 @@ ] } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.eids[0].uids[0] missing required field: \"id\"\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-empty.json index 44bee775844..7ec84992778 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.prebid requires a \"buyeruids\" property with at least one ID defined. If none exist, then request.user.ext.prebid should not be defined.\n", - "requestPayload": { + "description": "Bid request with an empty request.user.ext.prebid.buyeruids object", + "mockBidRequest": { "id": "request-without-user-ext-obj", "site": { "page": "test.somepage.com" @@ -30,5 +30,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.prebid requires a \"buyeruids\" property with at least one ID defined. If none exist, then request.user.ext.prebid should not be defined.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-unknown.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-unknown.json index 78773066744..715cc667274 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-unknown.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-buyeruids-unknown.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.unknown is neither a known bidder name nor an alias in request.ext.prebid.aliases.\n", - "requestPayload": { + "description": "Bid request with an unknown bidder name or alias inside the request.user.ext.unknown object", + "mockBidRequest": { "id": "request-without-user-ext-obj", "site": { "page": "test.somepage.com" @@ -32,5 +32,7 @@ } } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.unknown is neither a known bidder name nor an alias in request.ext.prebid.aliases.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-empty.json index f2e497514b7..6d811cf7030 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-ext-prebid-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.user.ext.prebid requires a \"buyeruids\" property with at least one ID defined. If none exist, then request.user.ext.prebid should not be defined.\n", - "requestPayload": { + "description": "Bid request with an empty request.user.ext.prebid object", + "mockBidRequest": { "id": "request-without-user-ext-obj", "site": { "page": "test.somepage.com" @@ -28,5 +28,7 @@ "prebid": {} } } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext.prebid requires a \"buyeruids\" property with at least one ID defined. If none exist, then request.user.ext.prebid should not be defined.\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-badtype.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-badtype.json deleted file mode 100644 index ce887889034..00000000000 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-badtype.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "message": "Invalid request: request.regs.ext is invalid: json: cannot unmarshal string into Go struct field ExtRegs.gdpr of type int8\n", - "requestPayload": { - "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", - "site": { - "page": "prebid.org", - "publisher": { - "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" - } - }, - "source": { - "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" - }, - "tmax": 1000, - "imp": [ - { - "id": "/19968336/header-bid-tag-0", - "ext": { - "appnexus": { - "placementId": 12883451 - } - }, - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 300 - } - ] - } - } - ], - "regs": { - "ext": { - "gdpr": "foo" - } - }, - "user": { - "ext": {} - } - } -} diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-invalid.json b/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-consent-invalid.json similarity index 70% rename from endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-invalid.json rename to endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-consent-invalid.json index 0729a22db80..08eed44b2b0 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-invalid.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/user-gdpr-consent-invalid.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.regs.ext.gdpr must be either 0 or 1.\n", - "requestPayload": { + "description": "Invalid GDPR consent string in user field", + "mockBidRequest": { "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", "site": { "page": "prebid.org", @@ -34,13 +34,12 @@ } } ], - "regs": { + "user": { "ext": { - "gdpr": 2 + "consent": 2 } - }, - "user": { - "ext": {} } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.user.ext object is not valid: json: cannot unmarshal number into Go struct field ExtUser.consent of type string" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/video-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/video-empty.json index 30fd1f13245..ec203d0f898 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/video-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/video-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].video.mimes must contain at least one supported MIME type\n", - "requestPayload": { + "description": "Bid request's sole imp element has empty video field and does not define a Banner, Video nor Audio field", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -11,5 +11,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].video.mimes must contain at least one supported MIME type\n" } diff --git a/endpoints/openrtb2/sample-requests/invalid-whole/video-mimes-empty.json b/endpoints/openrtb2/sample-requests/invalid-whole/video-mimes-empty.json index 5eb5c36f514..d2152dd29ed 100644 --- a/endpoints/openrtb2/sample-requests/invalid-whole/video-mimes-empty.json +++ b/endpoints/openrtb2/sample-requests/invalid-whole/video-mimes-empty.json @@ -1,6 +1,6 @@ { - "message": "Invalid request: request.imp[0].video.mimes must contain at least one supported MIME type\n", - "requestPayload": { + "description": "Bid request with empty mimes array in a video imp element", + "mockBidRequest": { "id": "req-id", "imp": [ { @@ -13,5 +13,7 @@ "app": { "id": "app_001" } - } + }, + "expectedReturnCode": 400, + "expectedErrorMessage": "Invalid request: request.imp[0].video.mimes must contain at least one supported MIME type\n" } diff --git a/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-hmin.json b/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-hmin.json index b20b461646c..15af8551da6 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-hmin.json +++ b/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-hmin.json @@ -1,11 +1,41 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ + "description": "Well formed native request that defines a 'wmin' on its 'img' field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request":"{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"wmin\":30}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ { - "img": { - "wmin": 30 + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 } + ], + "seat": "appnexus-bids" } - ] -} \ No newline at end of file + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-wmin.json b/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-wmin.json index f34dca050a6..5d986bcf755 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-wmin.json +++ b/endpoints/openrtb2/sample-requests/valid-native/asset-img-no-wmin.json @@ -1,11 +1,41 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ + "description": "Well formed native request that defines a 'hmin' on its 'img' field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"hmin\":30}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ { - "img": { - "hmin": 30 + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 } + ], + "seat": "appnexus-bids" } - ] -} \ No newline at end of file + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-native/asset-with-id.json b/endpoints/openrtb2/sample-requests/valid-native/asset-with-id.json index 7c55888ba29..1e55cdda63f 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/asset-with-id.json +++ b/endpoints/openrtb2/sample-requests/valid-native/asset-with-id.json @@ -1,12 +1,41 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ + "description": "Well formed native request that comes with 'id' field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"id\":1,\"img\":{\"wmin\":30}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ { - "id": 1, - "img": { - "wmin": 30 + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 } + ], + "seat": "appnexus-bids" } - ] + ] + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-native/asset-with-no-id.json b/endpoints/openrtb2/sample-requests/valid-native/asset-with-no-id.json index a9bc57ea274..36a1745cb19 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/asset-with-no-id.json +++ b/endpoints/openrtb2/sample-requests/valid-native/asset-with-no-id.json @@ -1,11 +1,41 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "img": { - "wmin": 30 - } + "description": "Well formed native request that comes with no 'id' field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"img\":{\"wmin\":30}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-native/assets-with-unique-ids.json b/endpoints/openrtb2/sample-requests/valid-native/assets-with-unique-ids.json index 10692b9aaf2..98cdeedadbe 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/assets-with-unique-ids.json +++ b/endpoints/openrtb2/sample-requests/valid-native/assets-with-unique-ids.json @@ -1,18 +1,41 @@ { - "context": 1, - "plcmttype": 1, - "assets": [ - { - "id": 1, - "img": { - "wmin": 30 - } + "description": "Multi-asset native request with different ids", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"id\":1,\"img\":{\"wmin\":30}},{\"id\":2,\"title\":{\"len\":20}}]}" }, - { - "id": 2, - "title": { - "len": 20 - } + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-native/request-no-context.json b/endpoints/openrtb2/sample-requests/valid-native/request-no-context.json index f5b0b35979e..1ad97c8ff8f 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/request-no-context.json +++ b/endpoints/openrtb2/sample-requests/valid-native/request-no-context.json @@ -1,11 +1,41 @@ { - "plcmttype": 2, - "assets": [ - { - "img": { - "hmin": 30, - "wmin": 20 - } + "description": "Well formed native request that comes with no 'context' field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"plcmttype\":2,\"assets\":[{\"img\":{\"hmin\":30,\"wmin\":20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-native/request-plcmttype-empty.json b/endpoints/openrtb2/sample-requests/valid-native/request-plcmttype-empty.json index 20dcdfb2aaa..88af803684d 100644 --- a/endpoints/openrtb2/sample-requests/valid-native/request-plcmttype-empty.json +++ b/endpoints/openrtb2/sample-requests/valid-native/request-plcmttype-empty.json @@ -1,11 +1,41 @@ { - "context": 1, - "assets": [ - { - "img": { - "hmin": 30, - "wmin": 20 - } + "description": "Well formed native request that comes with no 'plcmttype' field", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"assets\":[{\"img\":{\"hmin\":30,\"wmin\":20}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } + } ] -} \ No newline at end of file + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-native/sample-v1.1.json b/endpoints/openrtb2/sample-requests/valid-native/sample-v1.1.json deleted file mode 100644 index c0011724adb..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-native/sample-v1.1.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "context": 1, - "contextsubtype": 10, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } - }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } - } - ] -} diff --git a/endpoints/openrtb2/sample-requests/valid-native/sample-v1.2.json b/endpoints/openrtb2/sample-requests/valid-native/sample-v1.2.json deleted file mode 100644 index 10e4e7fdf61..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-native/sample-v1.2.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "context": 1, - "plcmttype": 1, - "assets": [ - { - "title": { - "len": 90 - } - }, - { - "img": { - "hmin": 30, - "wmin": 20 - } - }, - { - "video": { - "mimes": ["video/mp4"], - "minduration": 5, - "maxduration": 10, - "protocols": [1] - } - }, - { - "data": { - "type": 2 - } - } - ], - "eventtrackers": [{ - "event": 1, - "methods": [1] - }] -} diff --git a/endpoints/openrtb2/sample-requests/valid-native/video-asset-event-tracker.json b/endpoints/openrtb2/sample-requests/valid-native/video-asset-event-tracker.json new file mode 100644 index 00000000000..ab192e14881 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/valid-native/video-asset-event-tracker.json @@ -0,0 +1,41 @@ +{ + "description": "Well formed native request with video asset", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}],\"eventtrackers\":[{\"event\":1,\"methods\":[1]}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-native/with-video-asset.json b/endpoints/openrtb2/sample-requests/valid-native/with-video-asset.json new file mode 100644 index 00000000000..0ec3c993251 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/valid-native/with-video-asset.json @@ -0,0 +1,41 @@ +{ + "description": "Well formed native request with video asset", + "mockBidRequest": { + "id": "req-id", + "site": { + "page": "some.page.com" + }, + "tmax": 500, + "imp": [ + { + "id": "some-imp", + "native": { + "request": "{\"context\":1,\"contextsubtype\":10,\"plcmttype\":1,\"assets\":[{\"title\":{\"len\":90}},{\"img\":{\"hmin\":30,\"wmin\":20}},{\"video\":{\"mimes\":[\"video/mp4\"],\"minduration\":5,\"maxduration\":10,\"protocols\":[1]}},{\"data\":{\"type\":2}}]}" + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ] + }, + "expectedBidResponse": { + "id": "req-id", + "bidid": "test bid id", + "nbr": 0, + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ] + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/exemplary/all-ext.json b/endpoints/openrtb2/sample-requests/valid-whole/exemplary/all-ext.json index 3e2beedefac..f875fa880bc 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/exemplary/all-ext.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/exemplary/all-ext.json @@ -1,7 +1,6 @@ { "description": "This demonstrates all of the OpenRTB extensions supported by Prebid Server. Very few requests will need all of these at once.", - - "requestPayload": { + "mockBidRequest": { "id": "some-request-id", "site": { "page": "prebid.org" @@ -80,5 +79,43 @@ } } } - } + }, + "expectedBidResponse": { + "id":"some-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + }, + { + "bid": [ + { + "id": "districtm-bid", + "impid": "", + "price": 0 + } + ], + "seat": "districtm-bids" + }, + { + "bid": [ + { + "id": "rubicon-bid", + "impid": "", + "price": 0 + } + ], + "seat": "rubicon-bids" + } + ], + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/exemplary/prebid-test-ad.json b/endpoints/openrtb2/sample-requests/valid-whole/exemplary/prebid-test-ad.json index fc4794328a4..2c6a34f569e 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/exemplary/prebid-test-ad.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/exemplary/prebid-test-ad.json @@ -1,7 +1,6 @@ { "description": "This uses Appnexus to fetch the prebid sample ad, as seen on prebid.org.", - - "requestPayload": { + "mockBidRequest": { "id": "some-request-id", "site": { "page": "prebid.org" @@ -29,5 +28,23 @@ } ], "tmax": 500 - } + }, + "expectedBidResponse": { + "id": "some-request-id", + "seatbid": [ + { + "bid": [ + { + "id": "appnexus-bid", + "impid": "", + "price": 0 + } + ], + "seat": "appnexus-bids" + } + ], + "bidid": "test bid id", + "nbr": 0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliased-buyeruids.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliased-buyeruids.json index 82125592e46..141b643a520 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliased-buyeruids.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliased-buyeruids.json @@ -1,40 +1,49 @@ { - "id": "request-without-user-ext-obj", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, + "description": "Well formed amp request that comes with user field and buyeruids values", + "mockBidRequest": { + "id": "request-without-user-ext-obj", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "user": { + "ext": { + "prebid": { + "buyeruids": { + "unknown": "123" + } + } + } + }, "ext": { - "appnexus": { - "placementId": 12883451 + "prebid": { + "aliases": { + "unknown": "appnexus" + } + } } - } - } - ], - "user": { - "ext": { - "prebid": { - "buyeruids": { - "unknown": "123" - } - } - } - }, - "ext": { - "prebid": { - "aliases": { - "unknown": "appnexus" - } - } - } + }, + "expectedBidResponse": { + "id":"request-without-user-ext-obj", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliases.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliases.json index f6137e4a019..be7c2269331 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliases.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/aliases.json @@ -1,28 +1,37 @@ { - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "video": { - "mimes": [ - "video/mp4" - ] - }, - "ext": { - "unknown": { - "placementId": 12883451 + "description": "Well formed amp request with aliases that should run properly", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "unknown": { + "placementId": 12883451 + } } } - } - ], - "ext": { - "prebid": { - "aliases": { - "unknown": "appnexus" + ], + "ext": { + "prebid": { + "aliases": { + "unknown": "appnexus" + } } } - } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/app.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/app.json deleted file mode 100644 index 66e05d7636b..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/app.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "id": "some-request-id", - "app": { - "ext": { - "prebid": { - "source": "prebid-mobile", - "version": "1.0.0" - } - } - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ] -} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/bid-adjustments.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/bid-adjustments.json deleted file mode 100644 index 0cf52a8915f..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/bid-adjustments.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "video": { - "mimes": [ - "video/mp4" - ] - }, - "ext": { - "unknown": { - "placementId": 12883451 - } - } - } - ], - "ext": { - "prebid": { - "bidadjustmentfactors": { - "appnexus": 2.0, - "unknown": 1.5 - }, - "aliases": { - "unknown": "appnexus" - } - } - } -} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-bids.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-bids.json deleted file mode 100644 index a4c93b3d3cb..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-bids.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "video": { - "mimes": [ - "video/mp4" - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "ext": { - "prebid": { - "cache": { - "bids": {} - }, - "targeting": {} - } - } -} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-vast.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-vast.json deleted file mode 100644 index fe9445358ba..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/cache-vast.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "video": { - "mimes": [ - "video/mp4" - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "ext": { - "prebid": { - "cache": { - "vastxml": {} - } - } - } -} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/ccpa-invalid.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/ccpa-invalid.json deleted file mode 100644 index f3b677635c0..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/ccpa-invalid.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", - "site": { - "page": "prebid.org", - "publisher": { - "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" - } - }, - "source": { - "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" - }, - "tmax": 1000, - "imp": [ - { - "id": "/19968336/header-bid-tag-0", - "ext": { - "appnexus": { - "placementId": 12883451 - } - }, - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 300 - } - ] - } - } - ], - "regs": { - "ext": { - "us_privacy": "invalid by length. allowed since it only produces a warning." - } - } - } - \ No newline at end of file diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/digitrust.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/digitrust.json index ca8e090760d..5cd070745ab 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/digitrust.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/digitrust.json @@ -1,41 +1,50 @@ { - "id": "request-with-valid-digitrust-obj", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" + "description": "Well formed amp request with digitrust extension that should run properly", + "mockBidRequest": { + "id": "request-with-valid-digitrust-obj", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 } } - } - ], - "user": { - "yob": 1989, - "ext": { - "digitrust": { - "id": "sample-digitrust-id", - "keyv": 1, - "pref": 0 + ], + "user": { + "yob": 1989, + "ext": { + "digitrust": { + "id": "sample-digitrust-id", + "keyv": 1, + "pref": 0 + } } } - } + }, + "expectedBidResponse": { + "id":"request-with-valid-digitrust-obj", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr-no-consentstring.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr-no-consentstring.json index 5a63c6d11ce..6922dfb2a5c 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr-no-consentstring.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr-no-consentstring.json @@ -1,43 +1,52 @@ { - "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", - "site": { - "page": "prebid.org", - "publisher": { - "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" - } - }, - "source": { - "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" - }, - "tmax": 1000, - "imp": [ - { - "id": "/19968336/header-bid-tag-0", - "ext": { - "appnexus": { - "placementId": 12883451 - } - }, - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 300 + "description": "Well formed amp request with GDPR value but missing consent string", + "mockBidRequest": { + "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", + "site": { + "page": "prebid.org", + "publisher": { + "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" + } + }, + "source": { + "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" + }, + "tmax": 1000, + "imp": [ + { + "id": "/19968336/header-bid-tag-0", + "ext": { + "appnexus": { + "placementId": 12883451 } - ] + }, + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 300 + } + ] + } } + ], + "regs": { + "ext": { + "gdpr": 1 + } + }, + "user": { + "ext": {} } - ], - "regs": { - "ext": { - "gdpr": 1 - } }, - "user": { - "ext": {} - } + "expectedBidResponse": { + "id":"b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr.json index ef9f10d0cd0..1e3a2d41f2c 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/gdpr.json @@ -1,45 +1,54 @@ { - "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", - "site": { - "page": "prebid.org", - "publisher": { - "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" - } - }, - "source": { - "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" - }, - "tmax": 1000, - "imp": [ - { - "id": "/19968336/header-bid-tag-0", - "ext": { - "appnexus": { - "placementId": 12883451 - } - }, - "banner": { - "format": [ - { - "w": 300, - "h": 250 - }, - { - "w": 300, - "h": 300 + "description": "Well formed amp request with GDPR value and consent string", + "mockBidRequest": { + "id": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", + "site": { + "page": "prebid.org", + "publisher": { + "id": "a3de7af2-a86a-4043-a77b-c7e86744155e" + } + }, + "source": { + "tid": "b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5" + }, + "tmax": 1000, + "imp": [ + { + "id": "/19968336/header-bid-tag-0", + "ext": { + "appnexus": { + "placementId": 12883451 } - ] + }, + "banner": { + "format": [ + { + "w": 300, + "h": 250 + }, + { + "w": 300, + "h": 300 + } + ] + } + } + ], + "regs": { + "ext": { + "gdpr": 1 + } + }, + "user": { + "ext": { + "consent": "some-consent-string" } - } - ], - "regs": { - "ext": { - "gdpr": 1 } }, - "user": { - "ext": { - "consent": "some-consent-string" - } - } + "expectedBidResponse": { + "id":"b9c97a4b-cbc4-483d-b2c4-58a19ed5cfc5", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-device-only.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-device-only.json deleted file mode 100644 index 64146eaebe8..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-device-only.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": {}, - "instl": 1, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "device": { - "h": 640, - "w": 320, - "ext": { - "prebid": { - "interstitial": { - "minwidthperc": 60, - "minheightperc": 60 - } - } - } - }, - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} - } - } - } -} \ No newline at end of file diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-no-extension.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-no-extension.json deleted file mode 100644 index 15cd832053f..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial-no-extension.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "instl": 1, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} - } - } - } -} \ No newline at end of file diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial.json deleted file mode 100644 index 64fc2fe2653..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/interstitial.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "instl": 1, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ], - "device":{ - "h": 640, - "w": 320, - "ext": { - "prebid": { - "interstitial": { - "minwidthperc": 60, - "minheightperc": 60 - } - } - } - }, - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} - } - } - } -} - \ No newline at end of file diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-amp.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-amp.json index 30c0afc800a..5bea908b41f 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-amp.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-amp.json @@ -1,44 +1,53 @@ { - "id": "some-request-id", - "site": { - "page": "test.somepage.com", - "ext": { - "amp": 1 - } - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" - } - ] - }, + "description": "Request that comes with a valid amp value in its site.ext field", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com", "ext": { - "appnexus": { - "placementId": 12883451 + "amp": 1 + } + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } } } - } - ], - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} + ], + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} + } } } - } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-dnt.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-dnt.json new file mode 100644 index 00000000000..fb9d3bff0f5 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-dnt.json @@ -0,0 +1,53 @@ +{ + "description": "Request that comes with a valid device and dnt fields", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "device": { + "dnt": 1 + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + } + ], + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} + } + } + } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 + } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv4.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv4.json new file mode 100644 index 00000000000..8ef362458b6 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv4.json @@ -0,0 +1,47 @@ +{ + "description": "Well formed request with valid IPV4 in its device field", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 600 + }] + }, + "pmp": { + "deals": [{ + "id": "some-deal-id" + }] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + }], + "device": { + "ip": "8.8.8.8" + }, + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} + } + } + } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv6.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv6.json new file mode 100644 index 00000000000..90610a3b5a0 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site-has-ipv6.json @@ -0,0 +1,47 @@ +{ + "description": "Well formed request with valid IPV6 in its device field", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "banner": { + "format": [{ + "w": 300, + "h": 600 + }] + }, + "pmp": { + "deals": [{ + "id": "some-deal-id" + }] + }, + "ext": { + "appnexus": { + "placementId": 12883451 + } + } + }], + "device": { + "ipv6": "8888::" + }, + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} + } + } + } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 +} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site.json index 7a25249c763..60b8d7ecd4f 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/site.json @@ -1,41 +1,50 @@ { - "id": "some-request-id", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" + "description": "Well formed amp request with valid Site field", + "mockBidRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 } } - } - ], - "ext": { - "prebid": { - "targeting": { - "pricegranularity": "low" - }, - "cache": { - "bids": {} + ], + "ext": { + "prebid": { + "targeting": { + "pricegranularity": "low" + }, + "cache": { + "bids": {} + } } } - } + }, + "expectedBidResponse": { + "id":"some-request-id", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/timeout.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/timeout.json deleted file mode 100644 index b3dbe1f5d4b..00000000000 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/timeout.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "id": "some-request-id", - "app": {}, - "tmax": 500, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" - } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 - } - } - } - ] -} diff --git a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/user.json b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/user.json index 243b0739b7b..686489cf1ff 100644 --- a/endpoints/openrtb2/sample-requests/valid-whole/supplementary/user.json +++ b/endpoints/openrtb2/sample-requests/valid-whole/supplementary/user.json @@ -1,34 +1,43 @@ { - "id": "request-without-user-ext-obj", - "site": { - "page": "test.somepage.com" - }, - "imp": [ - { - "id": "my-imp-id", - "banner": { - "format": [ - { - "w": 300, - "h": 600 - } - ] - }, - "pmp": { - "deals": [ - { - "id": "some-deal-id" + "description": "Well formed amp request with valid User field", + "mockBidRequest": { + "id": "request-without-user-ext-obj", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "banner": { + "format": [ + { + "w": 300, + "h": 600 + } + ] + }, + "pmp": { + "deals": [ + { + "id": "some-deal-id" + } + ] + }, + "ext": { + "appnexus": { + "placementId": 12883451 } - ] - }, - "ext": { - "appnexus": { - "placementId": 12883451 } } + ], + "user": { + "yob": 1989 } - ], - "user": { - "yob": 1989 - } + }, + "expectedBidResponse": { + "id":"request-without-user-ext-obj", + "bidid":"test bid id", + "nbr":0 + }, + "expectedReturnCode": 200 } diff --git a/endpoints/openrtb2/sample-requests/video/video_valid_sample_appendbiddernames.json b/endpoints/openrtb2/sample-requests/video/video_valid_sample_appendbiddernames.json new file mode 100644 index 00000000000..4850fe91652 --- /dev/null +++ b/endpoints/openrtb2/sample-requests/video/video_valid_sample_appendbiddernames.json @@ -0,0 +1,86 @@ +{ + "description": "Video endpoint valid request with AppendBidderNames.", + "requestPayload": { + "appendbiddernames": true, + "storedrequestid": "80ce30c53c16e6ede735f123ef6e32361bfc7b22", + "podconfig": { + "durationrangesec": [ + 30 + ], + "requireexactduration": true, + "pods": [{ + "podid": 1, + "adpoddurationsec": 180, + "configid": "fba10607-0c12-43d1-ad07-b8a513bc75d6" + }, + { + "podid": 2, + "adpoddurationsec": 150, + "configid": "8b452b41-2681-4a20-9086-6f16ffad7773" + } + ] + }, + "site": { + "page": "prebid.com" + }, + "regs": { + "ext": { + "gdpr": 0 + } + }, + "user": { + "yob": 1991, + "gender": "F", + "keywords": "Hotels, Travelling", + "ext": { + "prebid": { + "buyeruids": { + "appnexus": "unique_id_an", + "rubicon": "unique_id_rubi" + } + } + } + }, + "device": { + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2", + "ip": "123.145.167.10", + "devicetype": 1, + "ifa": "AA000DFE74168477C70D291f574D344790E0BB11", + "lmt": 44, + "os": "mac os", + "w": 640, + "h": 480, + "didsha1": "didsha1", + "didmd5": "didmd5", + "dpidsha1": "dpidsha1", + "dpidmd5": "dpidmd5", + "macsha1": "macsha1", + "macmd5": "macmd5" + }, + "includebrandcategory": { + "primaryadserver": 1, + "publisher": "" + }, + "video": { + "w": 640, + "h": 480, + "mimes": [ + "video/mp4" + ], + "protocols": [ + 2, 3, 5, 6 + ] + }, + "content": { + "episode": 6, + "title": "episodeName", + "series": "TvName", + "season": "season3", + "len": 900, + "livestream": 0 + }, + "cacheconfig": { + "ttl": 42 + } + } +} \ No newline at end of file diff --git a/endpoints/openrtb2/video_auction.go b/endpoints/openrtb2/video_auction.go index 2629eb24454..2e806bffc07 100644 --- a/endpoints/openrtb2/video_auction.go +++ b/endpoints/openrtb2/video_auction.go @@ -15,11 +15,13 @@ import ( "time" "github.com/PubMatic-OpenWrap/prebid-server/errortypes" + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" "github.com/buger/jsonparser" jsonpatch "github.com/evanphx/json-patch" "github.com/gofrs/uuid" "github.com/PubMatic-OpenWrap/openrtb" + accountService "github.com/PubMatic-OpenWrap/prebid-server/account" "github.com/PubMatic-OpenWrap/prebid-server/analytics" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/exchange" @@ -34,16 +36,37 @@ import ( var defaultRequestTimeout int64 = 5000 -func NewVideoEndpoint(ex exchange.Exchange, validator openrtb_ext.BidderParamValidator, requestsById stored_requests.Fetcher, videoFetcher stored_requests.Fetcher, categories stored_requests.CategoryFetcher, cfg *config.Configuration, met pbsmetrics.MetricsEngine, pbsAnalytics analytics.PBSAnalyticsModule, disabledBidders map[string]string, defReqJSON []byte, bidderMap map[string]openrtb_ext.BidderName, cache prebid_cache_client.Client) (httprouter.Handle, error) { +func NewVideoEndpoint(ex exchange.Exchange, validator openrtb_ext.BidderParamValidator, requestsById stored_requests.Fetcher, videoFetcher stored_requests.Fetcher, accounts stored_requests.AccountFetcher, cfg *config.Configuration, met pbsmetrics.MetricsEngine, pbsAnalytics analytics.PBSAnalyticsModule, disabledBidders map[string]string, defReqJSON []byte, bidderMap map[string]openrtb_ext.BidderName, cache prebid_cache_client.Client) (httprouter.Handle, error) { - if ex == nil || validator == nil || requestsById == nil || cfg == nil || met == nil { + if ex == nil || validator == nil || requestsById == nil || accounts == nil || cfg == nil || met == nil { return nil, errors.New("NewVideoEndpoint requires non-nil arguments.") } + defRequest := defReqJSON != nil && len(defReqJSON) > 0 + ipValidator := iputil.PublicNetworkIPValidator{ + IPv4PrivateNetworks: cfg.RequestValidation.IPv4PrivateNetworksParsed, + IPv6PrivateNetworks: cfg.RequestValidation.IPv6PrivateNetworksParsed, + } + videoEndpointRegexp := regexp.MustCompile(`[<>]`) - return httprouter.Handle((&endpointDeps{ex, validator, requestsById, videoFetcher, categories, cfg, met, pbsAnalytics, disabledBidders, defRequest, defReqJSON, bidderMap, cache, videoEndpointRegexp}).VideoAuctionEndpoint), nil + return httprouter.Handle((&endpointDeps{ + ex, + validator, + requestsById, + videoFetcher, + accounts, + cfg, + met, + pbsAnalytics, + disabledBidders, + defRequest, + defReqJSON, + bidderMap, + cache, + videoEndpointRegexp, + ipValidator}).VideoAuctionEndpoint), nil } /* @@ -100,7 +123,7 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re defer func() { if len(debugLog.CacheKey) > 0 && vo.VideoResponse == nil { - err := putDebugLogError(deps.cache, &debugLog, start) + err := debugLog.PutDebugLogError(deps.cache, deps.cfg.CacheURL.ExpectedTimeMillis, vo.Errors) if err != nil { vo.Errors = append(vo.Errors, err) } @@ -220,7 +243,7 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re usersyncs := usersync.ParsePBSCookieFromRequest(r, &(deps.cfg.HostCookie)) if bidReq.App != nil { labels.Source = pbsmetrics.DemandApp - labels.PubID = effectivePubID(bidReq.App.Publisher) + labels.PubID = getAccountID(bidReq.App.Publisher) } else { // both bidReq.App == nil and bidReq.Site != nil are true labels.Source = pbsmetrics.DemandWeb if usersyncs.LiveSyncCount() == 0 { @@ -228,16 +251,25 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re } else { labels.CookieFlag = pbsmetrics.CookieFlagYes } - labels.PubID = effectivePubID(bidReq.Site.Publisher) + labels.PubID = getAccountID(bidReq.Site.Publisher) } - if acctIdErr := validateAccount(deps.cfg, labels.PubID); acctIdErr != nil { - errL := []error{err} - handleError(&labels, w, errL, &vo, &debugLog) + // Look up account now that we have resolved the pubID value + account, acctIDErrs := accountService.GetAccount(ctx, deps.cfg, deps.accounts, labels.PubID) + if len(acctIDErrs) > 0 { + handleError(&labels, w, acctIDErrs, &vo, &debugLog) return } - //execute auction logic - response, err := deps.ex.HoldAuction(ctx, bidReq, usersyncs, labels, &deps.categories, &debugLog) + + auctionRequest := exchange.AuctionRequest{ + BidRequest: bidReq, + Account: *account, + UserSyncs: usersyncs, + RequestType: labels.RType, + LegacyLabels: labels, + } + + response, err := deps.ex.HoldAuction(ctx, auctionRequest, &debugLog) vo.Request = bidReq vo.Response = response if err != nil { @@ -257,6 +289,21 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re bidResp.Ext = response.Ext } + if len(bidResp.AdPods) == 0 && debugLog.Enabled { + err := debugLog.PutDebugLogError(deps.cache, deps.cfg.CacheURL.ExpectedTimeMillis, vo.Errors) + if err != nil { + vo.Errors = append(vo.Errors, err) + } else { + bidResp.AdPods = append(bidResp.AdPods, &openrtb_ext.AdPod{ + Targeting: []openrtb_ext.VideoTargeting{ + { + HbCacheID: debugLog.CacheKey, + }, + }, + }) + } + } + vo.VideoResponse = bidResp resp, err := json.Marshal(bidResp) @@ -272,34 +319,6 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re } -func putDebugLogError(cache prebid_cache_client.Client, debugLog *exchange.DebugLog, start time.Time) error { - debugLog.Data.Response = "No response created" - - debugLog.BuildCacheString() - - data, err := json.Marshal(debugLog.CacheString) - if err != nil { - return err - } - - toCache := []prebid_cache_client.Cacheable{ - { - Type: debugLog.CacheType, - Data: data, - TTLSeconds: debugLog.TTL, - Key: "log_" + debugLog.CacheKey, - }, - } - - if cache != nil { - ctx, cancel := context.WithDeadline(context.Background(), start.Add(time.Duration(100)*time.Millisecond)) - defer cancel() - cache.PutJson(ctx, toCache) - } - - return nil -} - func cleanupVideoBidRequest(videoReq *openrtb_ext.BidRequestVideo, podErrors []PodError) *openrtb_ext.BidRequestVideo { for i := len(podErrors) - 1; i >= 0; i-- { videoReq.PodConfig.Pods = append(videoReq.PodConfig.Pods[:podErrors[i].PodIndex], videoReq.PodConfig.Pods[podErrors[i].PodIndex+1:]...) @@ -448,7 +467,7 @@ func buildVideoResponse(bidresponse *openrtb.BidResponse, podErrors []PodError) if err := json.Unmarshal(bid.Ext, &tempRespBidExt); err != nil { return nil, err } - if tempRespBidExt.Prebid.Targeting[string(openrtb_ext.HbVastCacheKey)] == "" { + if tempRespBidExt.Prebid.Targeting[formatTargetingKey(openrtb_ext.HbVastCacheKey, seatBid.Seat)] == "" { continue } @@ -457,9 +476,9 @@ func buildVideoResponse(bidresponse *openrtb.BidResponse, podErrors []PodError) podId, _ := strconv.ParseInt(podNum, 0, 64) videoTargeting := openrtb_ext.VideoTargeting{ - HbPb: tempRespBidExt.Prebid.Targeting[string(openrtb_ext.HbpbConstantKey)], - HbPbCatDur: tempRespBidExt.Prebid.Targeting[string(openrtb_ext.HbCategoryDurationKey)], - HbCacheID: tempRespBidExt.Prebid.Targeting[string(openrtb_ext.HbVastCacheKey)], + HbPb: tempRespBidExt.Prebid.Targeting[formatTargetingKey(openrtb_ext.HbpbConstantKey, seatBid.Seat)], + HbPbCatDur: tempRespBidExt.Prebid.Targeting[formatTargetingKey(openrtb_ext.HbCategoryDurationKey, seatBid.Seat)], + HbCacheID: tempRespBidExt.Prebid.Targeting[formatTargetingKey(openrtb_ext.HbVastCacheKey, seatBid.Seat)], } adPod := findAdPod(podId, adPods) @@ -497,6 +516,14 @@ func buildVideoResponse(bidresponse *openrtb.BidResponse, podErrors []PodError) return &openrtb_ext.BidResponseVideo{AdPods: adPods}, nil } +func formatTargetingKey(key openrtb_ext.TargetingKey, bidderName string) string { + fullKey := fmt.Sprintf("%s_%s", string(key), bidderName) + if len(fullKey) > exchange.MaxKeyLength { + return string(fullKey[0:exchange.MaxKeyLength]) + } + return fullKey +} + func findAdPod(podInd int64, pods []*openrtb_ext.AdPod) *openrtb_ext.AdPod { for _, pod := range pods { if pod.PodId == podInd { @@ -601,9 +628,10 @@ func createBidExtension(videoRequest *openrtb_ext.BidRequestVideo) ([]byte, erro targeting := openrtb_ext.ExtRequestTargeting{ PriceGranularity: priceGranularity, - IncludeWinners: true, IncludeBrandCategory: inclBrandCat, DurationRangeSec: durationRangeSec, + IncludeBidderKeys: true, + AppendBidderNames: videoRequest.AppendBidderNames, } vastXml := openrtb_ext.ExtRequestPrebidCacheVAST{} diff --git a/endpoints/openrtb2/video_auction_test.go b/endpoints/openrtb2/video_auction_test.go index c21b4324ba0..0e85d07c675 100644 --- a/endpoints/openrtb2/video_auction_test.go +++ b/endpoints/openrtb2/video_auction_test.go @@ -20,7 +20,6 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" - "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/empty_fetcher" metrics "github.com/rcrowley/go-metrics" "github.com/stretchr/testify/assert" @@ -81,6 +80,10 @@ func TestVideoEndpointImpressionsDuration(t *testing.T) { t.Fatalf("The request never made it into the Exchange.") } + var extData openrtb_ext.ExtRequest + json.Unmarshal(ex.lastRequest.Ext, &extData) + assert.True(t, extData.Prebid.Targeting.IncludeBidderKeys, "Request ext incorrect: IncludeBidderKeys should be true ") + assert.Len(t, ex.lastRequest.Imp, 22, "Incorrect number of impressions in request") assert.Equal(t, ex.lastRequest.Imp[0].ID, "1_0", "Incorrect impression id in request") assert.Equal(t, ex.lastRequest.Imp[0].Video.MaxDuration, int64(15), "Incorrect impression max duration in request") @@ -280,6 +283,42 @@ func TestVideoEndpointDebugError(t *testing.T) { assert.Equal(t, recorder.Code, 500, "Should catch error in request") } +func TestVideoEndpointDebugNoAdPods(t *testing.T) { + ex := &mockExchangeVideoNoBids{ + cache: &mockCacheClient{}, + } + reqData, err := ioutil.ReadFile("sample-requests/video/video_valid_sample.json") + if err != nil { + t.Fatalf("Failed to fetch a valid request: %v", err) + } + reqBody := string(getRequestPayload(t, reqData)) + req := httptest.NewRequest("POST", "/openrtb2/video?debug=true", strings.NewReader(reqBody)) + recorder := httptest.NewRecorder() + + deps := mockDepsNoBids(t, ex) + deps.VideoAuctionEndpoint(recorder, req, nil) + + if ex.lastRequest == nil { + t.Fatalf("The request never made it into the Exchange.") + } + if !ex.cache.called { + t.Fatalf("Cache was not called when it should have been") + } + + respBytes := recorder.Body.Bytes() + resp := &openrtb_ext.BidResponseVideo{} + if err := json.Unmarshal(respBytes, resp); err != nil { + t.Fatalf("Unable to unmarshal response.") + } + + assert.Len(t, resp.AdPods, 1, "Debug AdPod should be added to response") + assert.Empty(t, resp.AdPods[0].Errors, "AdPod Errors should be empty") + assert.Empty(t, resp.AdPods[0].Targeting[0].HbPb, "Hb_pb should be empty") + assert.Empty(t, resp.AdPods[0].Targeting[0].HbPbCatDur, "Hb_pb_cat_dur should be empty") + assert.NotEmpty(t, resp.AdPods[0].Targeting[0].HbCacheID, "Hb_cache_id should not be empty") + assert.Equal(t, int64(0), resp.AdPods[0].PodId, "Pod ID should be 0") +} + func TestVideoEndpointNoPods(t *testing.T) { ex := &mockExchangeVideo{} reqData, err := ioutil.ReadFile("sample-requests/video/video_invalid_sample.json") @@ -643,9 +682,9 @@ func TestVideoBuildVideoResponseMissedCacheForOneBid(t *testing.T) { bid2 := openrtb.Bid{} bid3 := openrtb.Bid{} - extBid1 := []byte(`{"prebid":{"targeting":{"hb_bidder":"appnexus","hb_pb":"17.00","hb_pb_cat_dur":"17.00_123_30s","hb_size":"1x1","hb_uuid":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) - extBid2 := []byte(`{"prebid":{"targeting":{"hb_bidder":"appnexus","hb_pb":"17.00","hb_pb_cat_dur":"17.00_456_30s","hb_size":"1x1","hb_uuid":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) - extBid3 := []byte(`{"prebid":{"targeting":{"hb_bidder":"appnexus","hb_pb":"17.00","hb_pb_cat_dur":"17.00_406_30s","hb_size":"1x1"}}}`) + extBid1 := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"17.00","hb_pb_cat_dur_appnex":"17.00_123_30s","hb_size":"1x1","hb_uuid_appnexus":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) + extBid2 := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"17.00","hb_pb_cat_dur_appnex":"17.00_456_30s","hb_size":"1x1","hb_uuid_appnexus":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) + extBid3 := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"17.00","hb_pb_cat_dur_appnex":"17.00_406_30s","hb_size":"1x1"}}}`) bid1.Ext = extBid1 bids = append(bids, bid1) @@ -657,6 +696,7 @@ func TestVideoBuildVideoResponseMissedCacheForOneBid(t *testing.T) { bids = append(bids, bid3) seatBid.Bid = bids + seatBid.Seat = "appnexus" seatBids = append(seatBids, seatBid) openRtbBidResp.SeatBid = seatBids @@ -713,8 +753,8 @@ func TestVideoBuildVideoResponsePodErrors(t *testing.T) { bid1 := openrtb.Bid{} bid2 := openrtb.Bid{} - extBid1 := []byte(`{"prebid":{"targeting":{"hb_bidder":"appnexus","hb_pb":"17.00","hb_pb_cat_dur":"17.00_123_30s","hb_size":"1x1","hb_uuid":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) - extBid2 := []byte(`{"prebid":{"targeting":{"hb_bidder":"appnexus","hb_pb":"17.00","hb_pb_cat_dur":"17.00_456_30s","hb_size":"1x1","hb_uuid":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) + extBid1 := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"17.00","hb_pb_cat_dur_appnex":"17.00_123_30s","hb_size":"1x1","hb_uuid_appnexus":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) + extBid2 := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"17.00","hb_pb_cat_dur_appnex":"17.00_456_30s","hb_size":"1x1","hb_uuid_appnexus":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"}}}`) bid1.Ext = extBid1 bids = append(bids, bid1) @@ -723,6 +763,7 @@ func TestVideoBuildVideoResponsePodErrors(t *testing.T) { bids = append(bids, bid2) seatBid.Bid = bids + seatBid.Seat = "appnexus" seatBids = append(seatBids, seatBid) openRtbBidResp.SeatBid = seatBids @@ -1107,10 +1148,62 @@ func TestCCPA(t *testing.T) { } } +func TestVideoEndpointAppendBidderNames(t *testing.T) { + ex := &mockExchangeAppendBidderNames{} + reqData, err := ioutil.ReadFile("sample-requests/video/video_valid_sample_appendbiddernames.json") + if err != nil { + t.Fatalf("Failed to fetch a valid request: %v", err) + } + reqBody := string(getRequestPayload(t, reqData)) + req := httptest.NewRequest("POST", "/openrtb2/video", strings.NewReader(reqBody)) + recorder := httptest.NewRecorder() + + deps := mockDepsAppendBidderNames(t, ex) + deps.VideoAuctionEndpoint(recorder, req, nil) + + if ex.lastRequest == nil { + t.Fatalf("The request never made it into the Exchange.") + } + + var extData openrtb_ext.ExtRequest + json.Unmarshal(ex.lastRequest.Ext, &extData) + assert.True(t, extData.Prebid.Targeting.AppendBidderNames, "Request ext incorrect: AppendBidderNames should be true ") + + respBytes := recorder.Body.Bytes() + resp := &openrtb_ext.BidResponseVideo{} + if err := json.Unmarshal(respBytes, resp); err != nil { + t.Fatalf("Unable to unmarshal response.") + } + + assert.Len(t, ex.lastRequest.Imp, 11, "Incorrect number of impressions in request") + assert.Equal(t, string(ex.lastRequest.Site.Page), "prebid.com", "Incorrect site page in request") + assert.Equal(t, ex.lastRequest.Site.Content.Series, "TvName", "Incorrect site content series in request") + + assert.Len(t, resp.AdPods, 5, "Incorrect number of Ad Pods in response") + assert.Len(t, resp.AdPods[0].Targeting, 4, "Incorrect Targeting data in response") + assert.Len(t, resp.AdPods[1].Targeting, 3, "Incorrect Targeting data in response") + assert.Len(t, resp.AdPods[2].Targeting, 5, "Incorrect Targeting data in response") + assert.Len(t, resp.AdPods[3].Targeting, 1, "Incorrect Targeting data in response") + assert.Len(t, resp.AdPods[4].Targeting, 3, "Incorrect Targeting data in response") + + assert.Equal(t, resp.AdPods[4].Targeting[0].HbPbCatDur, "20.00_395_30s_appnexus", "Incorrect number of Ad Pods in response") + +} + +func TestFormatTargetingKey(t *testing.T) { + res := formatTargetingKey(openrtb_ext.HbCategoryDurationKey, "appnexus") + assert.Equal(t, "hb_pb_cat_dur_appnex", res, "Tergeting key constructed incorrectly") +} + +func TestFormatTargetingKeyLongKey(t *testing.T) { + res := formatTargetingKey(openrtb_ext.HbpbConstantKey, "20.00") + assert.Equal(t, "hb_pb_20.00", res, "Tergeting key constructed incorrectly") +} + func mockDepsWithMetrics(t *testing.T, ex *mockExchangeVideo) (*endpointDeps, *pbsmetrics.Metrics, *mockAnalyticsModule) { theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) mockModule := &mockAnalyticsModule{} - edep := &endpointDeps{ + deps := &endpointDeps{ ex, newParamsValidator(t), &mockVideoStoredReqFetcher{}, @@ -1125,9 +1218,10 @@ func mockDepsWithMetrics(t *testing.T, ex *mockExchangeVideo) (*endpointDeps, *p openrtb_ext.BidderMap, nil, nil, + hardcodedResponseIPValidator{response: true}, } - return edep, theMetrics, mockModule + return deps, theMetrics, mockModule } type mockAnalyticsModule struct { @@ -1149,7 +1243,55 @@ func (m *mockAnalyticsModule) LogSetUIDObject(so *analytics.SetUIDObject) { retu func (m *mockAnalyticsModule) LogAmpObject(ao *analytics.AmpObject) { return } +func (m *mockAnalyticsModule) LogNotificationEventObject(ne *analytics.NotificationEvent) { return } + func mockDeps(t *testing.T, ex *mockExchangeVideo) *endpointDeps { + theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + deps := &endpointDeps{ + ex, + newParamsValidator(t), + &mockVideoStoredReqFetcher{}, + &mockVideoStoredReqFetcher{}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{MaxRequestSize: maxSize}, + theMetrics, + analyticsConf.NewPBSAnalytics(&config.Analytics{}), + map[string]string{}, + false, + []byte{}, + openrtb_ext.BidderMap, + ex.cache, + regexp.MustCompile(`[<>]`), + hardcodedResponseIPValidator{response: true}, + } + + return deps +} + +func mockDepsAppendBidderNames(t *testing.T, ex *mockExchangeAppendBidderNames) *endpointDeps { + theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) + deps := &endpointDeps{ + ex, + newParamsValidator(t), + &mockVideoStoredReqFetcher{}, + &mockVideoStoredReqFetcher{}, + empty_fetcher.EmptyFetcher{}, + &config.Configuration{MaxRequestSize: maxSize}, + theMetrics, + analyticsConf.NewPBSAnalytics(&config.Analytics{}), + map[string]string{}, + false, + []byte{}, + openrtb_ext.BidderMap, + ex.cache, + regexp.MustCompile(`[<>]`), + hardcodedResponseIPValidator{response: true}, + } + + return deps +} + +func mockDepsNoBids(t *testing.T, ex *mockExchangeVideoNoBids) *endpointDeps { theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) edep := &endpointDeps{ ex, @@ -1166,6 +1308,7 @@ func mockDeps(t *testing.T, ex *mockExchangeVideo) *endpointDeps { openrtb_ext.BidderMap, ex.cache, regexp.MustCompile(`[<>]`), + hardcodedResponseIPValidator{response: true}, } return edep @@ -1182,8 +1325,8 @@ func (m *mockCacheClient) PutJson(ctx context.Context, values []prebid_cache_cli return []string{}, []error{} } -func (m *mockCacheClient) GetExtCacheData() (string, string) { - return "", "" +func (m *mockCacheClient) GetExtCacheData() (scheme string, host string, path string) { + return "", "", "" } type mockVideoStoredReqFetcher struct { @@ -1198,14 +1341,15 @@ type mockExchangeVideo struct { cache *mockCacheClient } -func (m *mockExchangeVideo) HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, ids exchange.IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { - m.lastRequest = bidRequest +func (m *mockExchangeVideo) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + m.lastRequest = r.BidRequest if debugLog != nil && debugLog.Enabled { m.cache.called = true } - ext := []byte(`{"prebid":{"targeting":{"hb_bidder":"appnexus","hb_pb":"20.00","hb_pb_cat_dur":"20.00_395_30s","hb_size":"1x1", "hb_uuid":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"},"type":"video"},"bidder":{"appnexus":{"brand_id":1,"auction_id":7840037870526938650,"bidder_id":2,"bid_ad_type":1,"creative_info":{"video":{"duration":30,"mimes":["video\/mp4"]}}}}}`) + ext := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"20.00","hb_pb_cat_dur_appnex":"20.00_395_30s","hb_size":"1x1", "hb_uuid_appnexus":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"},"type":"video","dealpriority":0,"dealtiersatisfied":false},"bidder":{"appnexus":{"brand_id":1,"auction_id":7840037870526938650,"bidder_id":2,"bid_ad_type":1,"creative_info":{"video":{"duration":30,"mimes":["video\/mp4"]}}}}}`) return &openrtb.BidResponse{ SeatBid: []openrtb.SeatBid{{ + Seat: "appnexus", Bid: []openrtb.Bid{ {ID: "01", ImpID: "1_0", Ext: ext}, {ID: "02", ImpID: "1_1", Ext: ext}, @@ -1228,6 +1372,54 @@ func (m *mockExchangeVideo) HoldAuction(ctx context.Context, bidRequest *openrtb }, nil } +type mockExchangeAppendBidderNames struct { + lastRequest *openrtb.BidRequest + cache *mockCacheClient +} + +func (m *mockExchangeAppendBidderNames) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + m.lastRequest = r.BidRequest + if debugLog != nil && debugLog.Enabled { + m.cache.called = true + } + ext := []byte(`{"prebid":{"targeting":{"hb_bidder_appnexus":"appnexus","hb_pb_appnexus":"20.00","hb_pb_cat_dur_appnex":"20.00_395_30s_appnexus","hb_size":"1x1", "hb_uuid_appnexus":"837ea3b7-5598-4958-8c45-8e9ef2bf7cc1"},"type":"video"},"bidder":{"appnexus":{"brand_id":1,"auction_id":7840037870526938650,"bidder_id":2,"bid_ad_type":1,"creative_info":{"video":{"duration":30,"mimes":["video\/mp4"]}}}}}`) + return &openrtb.BidResponse{ + SeatBid: []openrtb.SeatBid{{ + Seat: "appnexus", + Bid: []openrtb.Bid{ + {ID: "01", ImpID: "1_0", Ext: ext}, + {ID: "02", ImpID: "1_1", Ext: ext}, + {ID: "03", ImpID: "1_2", Ext: ext}, + {ID: "04", ImpID: "1_3", Ext: ext}, + {ID: "05", ImpID: "2_0", Ext: ext}, + {ID: "06", ImpID: "2_1", Ext: ext}, + {ID: "07", ImpID: "2_2", Ext: ext}, + {ID: "08", ImpID: "3_0", Ext: ext}, + {ID: "09", ImpID: "3_1", Ext: ext}, + {ID: "10", ImpID: "3_2", Ext: ext}, + {ID: "11", ImpID: "3_3", Ext: ext}, + {ID: "12", ImpID: "3_5", Ext: ext}, + {ID: "13", ImpID: "4_0", Ext: ext}, + {ID: "14", ImpID: "5_0", Ext: ext}, + {ID: "15", ImpID: "5_1", Ext: ext}, + {ID: "16", ImpID: "5_2", Ext: ext}, + }, + }}, + }, nil +} + +type mockExchangeVideoNoBids struct { + lastRequest *openrtb.BidRequest + cache *mockCacheClient +} + +func (m *mockExchangeVideoNoBids) HoldAuction(ctx context.Context, r exchange.AuctionRequest, debugLog *exchange.DebugLog) (*openrtb.BidResponse, error) { + m.lastRequest = r.BidRequest + return &openrtb.BidResponse{ + SeatBid: []openrtb.SeatBid{{}}, + }, nil +} + var testVideoStoredImpData = map[string]json.RawMessage{ "fba10607-0c12-43d1-ad07-b8a513bc75d6": json.RawMessage(`{"ext": {"appnexus": {"placementId": 14997137}}}`), "8b452b41-2681-4a20-9086-6f16ffad7773": json.RawMessage(`{"ext": {"appnexus": {"placementId": 15016213}}}`), diff --git a/endpoints/setuid_test.go b/endpoints/setuid_test.go index 7b056d85f4b..d731b2dff17 100644 --- a/endpoints/setuid_test.go +++ b/endpoints/setuid_test.go @@ -443,8 +443,8 @@ func (g *mockPermsSetUID) BidderSyncAllowed(ctx context.Context, bidder openrtb_ return false, nil } -func (g *mockPermsSetUID) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) { - return g.allowPI, g.allowPI, nil +func (g *mockPermsSetUID) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { + return g.allowPI, g.allowPI, g.allowPI, nil } func (g *mockPermsSetUID) AMPException() bool { diff --git a/exchange/adapter_map.go b/exchange/adapter_map.go index c01dc64da52..b8855583a2a 100755 --- a/exchange/adapter_map.go +++ b/exchange/adapter_map.go @@ -9,26 +9,33 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/adapters" ttx "github.com/PubMatic-OpenWrap/prebid-server/adapters/33across" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/acuityads" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adform" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adgeneration" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adhese" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adkernel" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adkernelAdn" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adman" "github.com/PubMatic-OpenWrap/prebid-server/adapters/admixer" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adocean" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adoppler" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adpone" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adprime" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adtarget" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adtelligent" "github.com/PubMatic-OpenWrap/prebid-server/adapters/advangelists" "github.com/PubMatic-OpenWrap/prebid-server/adapters/aja" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/amx" "github.com/PubMatic-OpenWrap/prebid-server/adapters/applogy" "github.com/PubMatic-OpenWrap/prebid-server/adapters/appnexus" "github.com/PubMatic-OpenWrap/prebid-server/adapters/audienceNetwork" "github.com/PubMatic-OpenWrap/prebid-server/adapters/avocet" "github.com/PubMatic-OpenWrap/prebid-server/adapters/beachfront" "github.com/PubMatic-OpenWrap/prebid-server/adapters/beintoo" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/between" "github.com/PubMatic-OpenWrap/prebid-server/adapters/brightroll" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/colossus" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/connectad" "github.com/PubMatic-OpenWrap/prebid-server/adapters/consumable" "github.com/PubMatic-OpenWrap/prebid-server/adapters/conversant" "github.com/PubMatic-OpenWrap/prebid-server/adapters/cpmstar" @@ -42,17 +49,22 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/adapters/grid" "github.com/PubMatic-OpenWrap/prebid-server/adapters/gumgum" "github.com/PubMatic-OpenWrap/prebid-server/adapters/improvedigital" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/inmobi" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/invibes" "github.com/PubMatic-OpenWrap/prebid-server/adapters/ix" "github.com/PubMatic-OpenWrap/prebid-server/adapters/kidoz" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/krushmedia" "github.com/PubMatic-OpenWrap/prebid-server/adapters/kubient" "github.com/PubMatic-OpenWrap/prebid-server/adapters/lifestreet" "github.com/PubMatic-OpenWrap/prebid-server/adapters/lockerdome" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/logicad" "github.com/PubMatic-OpenWrap/prebid-server/adapters/lunamedia" "github.com/PubMatic-OpenWrap/prebid-server/adapters/marsmedia" "github.com/PubMatic-OpenWrap/prebid-server/adapters/mgid" "github.com/PubMatic-OpenWrap/prebid-server/adapters/mobilefuse" "github.com/PubMatic-OpenWrap/prebid-server/adapters/nanointeractive" "github.com/PubMatic-OpenWrap/prebid-server/adapters/ninthdecimal" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/nobid" "github.com/PubMatic-OpenWrap/prebid-server/adapters/openx" "github.com/PubMatic-OpenWrap/prebid-server/adapters/orbidder" "github.com/PubMatic-OpenWrap/prebid-server/adapters/pubmatic" @@ -62,7 +74,11 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/adapters/rtbhouse" "github.com/PubMatic-OpenWrap/prebid-server/adapters/rubicon" "github.com/PubMatic-OpenWrap/prebid-server/adapters/sharethrough" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/silvermob" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/smaato" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/smartadserver" "github.com/PubMatic-OpenWrap/prebid-server/adapters/smartrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/smartyads" "github.com/PubMatic-OpenWrap/prebid-server/adapters/somoaudience" "github.com/PubMatic-OpenWrap/prebid-server/adapters/sonobi" "github.com/PubMatic-OpenWrap/prebid-server/adapters/sovrn" @@ -93,26 +109,33 @@ import ( func newAdapterMap(client *http.Client, cfg *config.Configuration, infos adapters.BidderInfos, me pbsmetrics.MetricsEngine) map[openrtb_ext.BidderName]adaptedBidder { ortbBidders := map[openrtb_ext.BidderName]adapters.Bidder{ openrtb_ext.Bidder33Across: ttx.New33AcrossBidder(cfg.Adapters[string(openrtb_ext.Bidder33Across)].Endpoint), + openrtb_ext.BidderAcuityAds: acuityads.NewAcuityAdsBidder(cfg.Adapters[string(openrtb_ext.BidderAcuityAds)].Endpoint), openrtb_ext.BidderAdform: adform.NewAdformBidder(client, cfg.Adapters[string(openrtb_ext.BidderAdform)].Endpoint), openrtb_ext.BidderAdgeneration: adgeneration.NewAdgenerationAdapter(cfg.Adapters[string(openrtb_ext.BidderAdgeneration)].Endpoint), openrtb_ext.BidderAdhese: adhese.NewAdheseBidder(cfg.Adapters[string(openrtb_ext.BidderAdhese)].Endpoint), openrtb_ext.BidderAdkernel: adkernel.NewAdkernelAdapter(cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderAdkernel))].Endpoint), openrtb_ext.BidderAdkernelAdn: adkernelAdn.NewAdkernelAdnAdapter(cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderAdkernelAdn))].Endpoint), + openrtb_ext.BidderAdman: adman.NewAdmanBidder(cfg.Adapters[string(openrtb_ext.BidderAdman)].Endpoint), openrtb_ext.BidderAdmixer: admixer.NewAdmixerBidder(cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderAdmixer))].Endpoint), openrtb_ext.BidderAdOcean: adocean.NewAdOceanBidder(client, cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderAdOcean))].Endpoint), openrtb_ext.BidderAdoppler: adoppler.NewAdopplerBidder(cfg.Adapters[string(openrtb_ext.BidderAdoppler)].Endpoint), openrtb_ext.BidderAdpone: adpone.NewAdponeBidder(cfg.Adapters[string(openrtb_ext.BidderAdpone)].Endpoint), + openrtb_ext.BidderAdprime: adprime.NewAdprimeBidder(cfg.Adapters[string(openrtb_ext.BidderAdprime)].Endpoint), openrtb_ext.BidderAdtarget: adtarget.NewAdtargetBidder(cfg.Adapters[string(openrtb_ext.BidderAdtarget)].Endpoint), openrtb_ext.BidderAdtelligent: adtelligent.NewAdtelligentBidder(cfg.Adapters[string(openrtb_ext.BidderAdtelligent)].Endpoint), openrtb_ext.BidderAdvangelists: advangelists.NewAdvangelistsBidder(cfg.Adapters[string(openrtb_ext.BidderAdvangelists)].Endpoint), openrtb_ext.BidderAJA: aja.NewAJABidder(cfg.Adapters[string(openrtb_ext.BidderAJA)].Endpoint), + openrtb_ext.BidderAMX: amx.NewAMXBidder(cfg.Adapters[string(openrtb_ext.BidderAMX)].Endpoint), openrtb_ext.BidderApplogy: applogy.NewApplogyBidder(cfg.Adapters[string(openrtb_ext.BidderApplogy)].Endpoint), openrtb_ext.BidderAppnexus: appnexus.NewAppNexusBidder(client, cfg.Adapters[string(openrtb_ext.BidderAppnexus)].Endpoint, cfg.Adapters[string(openrtb_ext.BidderAppnexus)].PlatformID), openrtb_ext.BidderAvocet: avocet.NewAvocetAdapter(cfg.Adapters[string(openrtb_ext.BidderAvocet)].Endpoint), openrtb_ext.BidderBeachfront: beachfront.NewBeachfrontBidder(cfg.Adapters[string(openrtb_ext.BidderBeachfront)].Endpoint, cfg.Adapters[string(openrtb_ext.BidderBeachfront)].ExtraAdapterInfo), openrtb_ext.BidderBeintoo: beintoo.NewBeintooBidder(cfg.Adapters[string(openrtb_ext.BidderBeintoo)].Endpoint), - openrtb_ext.BidderBrightroll: brightroll.NewBrightrollBidder(cfg.Adapters[string(openrtb_ext.BidderBrightroll)].Endpoint), + openrtb_ext.BidderBrightroll: brightroll.NewBrightrollBidder(cfg.Adapters[string(openrtb_ext.BidderBrightroll)].Endpoint, cfg.Adapters[string(openrtb_ext.BidderBrightroll)].ExtraAdapterInfo), + openrtb_ext.BidderColossus: colossus.NewColossusBidder(cfg.Adapters[string(openrtb_ext.BidderColossus)].Endpoint), + openrtb_ext.BidderConnectAd: connectad.NewConnectAdBidder(cfg.Adapters[string(openrtb_ext.BidderConnectAd)].Endpoint), openrtb_ext.BidderConsumable: consumable.NewConsumableBidder(cfg.Adapters[string(openrtb_ext.BidderConsumable)].Endpoint), + openrtb_ext.BidderConversant: conversant.NewConversantBidder(cfg.Adapters[string(openrtb_ext.BidderConversant)].Endpoint), openrtb_ext.BidderCpmstar: cpmstar.NewCpmstarBidder(cfg.Adapters[string(openrtb_ext.BidderCpmstar)].Endpoint), openrtb_ext.BidderDatablocks: datablocks.NewDatablocksBidder(cfg.Adapters[string(openrtb_ext.BidderDatablocks)].Endpoint), openrtb_ext.BidderDmx: dmx.NewDmxBidder(cfg.Adapters[string(openrtb_ext.BidderDmx)].Endpoint), @@ -120,7 +143,6 @@ func newAdapterMap(client *http.Client, cfg *config.Configuration, infos adapter openrtb_ext.BidderEngageBDR: engagebdr.NewEngageBDRBidder(client, cfg.Adapters[string(openrtb_ext.BidderEngageBDR)].Endpoint), openrtb_ext.BidderEPlanning: eplanning.NewEPlanningBidder(client, cfg.Adapters[string(openrtb_ext.BidderEPlanning)].Endpoint), openrtb_ext.BidderFacebook: audienceNetwork.NewFacebookBidder( - client, cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderFacebook))].PlatformID, cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderFacebook))].AppSecret), openrtb_ext.BidderGamma: gamma.NewGammaBidder(cfg.Adapters[string(openrtb_ext.BidderGamma)].Endpoint), @@ -128,15 +150,20 @@ func newAdapterMap(client *http.Client, cfg *config.Configuration, infos adapter openrtb_ext.BidderGrid: grid.NewGridBidder(cfg.Adapters[string(openrtb_ext.BidderGrid)].Endpoint), openrtb_ext.BidderGumGum: gumgum.NewGumGumBidder(cfg.Adapters[string(openrtb_ext.BidderGumGum)].Endpoint), openrtb_ext.BidderImprovedigital: improvedigital.NewImprovedigitalBidder(cfg.Adapters[string(openrtb_ext.BidderImprovedigital)].Endpoint), + openrtb_ext.BidderInMobi: inmobi.NewInMobiAdapter(cfg.Adapters[string(openrtb_ext.BidderInMobi)].Endpoint), + openrtb_ext.BidderInvibes: invibes.NewInvibesBidder(cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderInvibes))].Endpoint), openrtb_ext.BidderKidoz: kidoz.NewKidozBidder(cfg.Adapters[string(openrtb_ext.BidderKidoz)].Endpoint), + openrtb_ext.BidderKrushmedia: krushmedia.NewKrushmediaBidder(cfg.Adapters[string(openrtb_ext.BidderKrushmedia)].Endpoint), openrtb_ext.BidderKubient: kubient.NewKubientBidder(cfg.Adapters[string(openrtb_ext.BidderKubient)].Endpoint), openrtb_ext.BidderLockerDome: lockerdome.NewLockerDomeBidder(cfg.Adapters[string(openrtb_ext.BidderLockerDome)].Endpoint), openrtb_ext.BidderLunaMedia: lunamedia.NewLunaMediaBidder(cfg.Adapters[string(openrtb_ext.BidderLunaMedia)].Endpoint), + openrtb_ext.BidderLogicad: logicad.NewLogicadBidder(cfg.Adapters[string(openrtb_ext.BidderLogicad)].Endpoint), openrtb_ext.BidderMarsmedia: marsmedia.NewMarsmediaBidder(cfg.Adapters[string(openrtb_ext.BidderMarsmedia)].Endpoint), openrtb_ext.BidderMgid: mgid.NewMgidBidder(cfg.Adapters[string(openrtb_ext.BidderMgid)].Endpoint), openrtb_ext.BidderMobileFuse: mobilefuse.NewMobileFuseBidder(cfg.Adapters[string(openrtb_ext.BidderMobileFuse)].Endpoint), openrtb_ext.BidderNanoInteractive: nanointeractive.NewNanoIneractiveBidder(cfg.Adapters[string(openrtb_ext.BidderNanoInteractive)].Endpoint), openrtb_ext.BidderNinthDecimal: ninthdecimal.NewNinthDecimalBidder(cfg.Adapters[string(openrtb_ext.BidderNinthDecimal)].Endpoint), + openrtb_ext.BidderNoBid: nobid.NewNoBidBidder(cfg.Adapters[string(openrtb_ext.BidderNoBid)].Endpoint), openrtb_ext.BidderOrbidder: orbidder.NewOrbidderBidder(cfg.Adapters[string(openrtb_ext.BidderOrbidder)].Endpoint), openrtb_ext.BidderOpenx: openx.NewOpenxBidder(cfg.Adapters[string(openrtb_ext.BidderOpenx)].Endpoint), openrtb_ext.BidderPubmatic: pubmatic.NewPubmaticBidder(client, cfg.Adapters[string(openrtb_ext.BidderPubmatic)].Endpoint), @@ -151,7 +178,11 @@ func newAdapterMap(client *http.Client, cfg *config.Configuration, infos adapter cfg.Adapters[string(openrtb_ext.BidderRubicon)].XAPI.Tracker), openrtb_ext.BidderSharethrough: sharethrough.NewSharethroughBidder(cfg.Adapters[string(openrtb_ext.BidderSharethrough)].Endpoint), + openrtb_ext.BidderSilverMob: silvermob.NewSilverMobBidder(cfg.Adapters[string(openrtb_ext.BidderSilverMob)].Endpoint), + openrtb_ext.BidderSmaato: smaato.NewSmaatoBidder(cfg.Adapters[string(openrtb_ext.BidderSmaato)].Endpoint), + openrtb_ext.BidderSmartadserver: smartadserver.NewSmartadserverBidder(cfg.Adapters[string(openrtb_ext.BidderSmartadserver)].Endpoint), openrtb_ext.BidderSmartRTB: smartrtb.NewSmartRTBBidder(cfg.Adapters[string(openrtb_ext.BidderSmartRTB)].Endpoint), + openrtb_ext.BidderSmartyAds: smartyads.NewSmartyAdsBidder(cfg.Adapters[string(openrtb_ext.BidderSmartyAds)].Endpoint), openrtb_ext.BidderSomoaudience: somoaudience.NewSomoaudienceBidder(cfg.Adapters[string(openrtb_ext.BidderSomoaudience)].Endpoint), openrtb_ext.BidderSonobi: sonobi.NewSonobiBidder(client, cfg.Adapters[string(openrtb_ext.BidderSonobi)].Endpoint), openrtb_ext.BidderSovrn: sovrn.NewSovrnBidder(client, cfg.Adapters[string(openrtb_ext.BidderSovrn)].Endpoint), @@ -172,11 +203,10 @@ func newAdapterMap(client *http.Client, cfg *config.Configuration, infos adapter openrtb_ext.BidderYieldmo: yieldmo.NewYieldmoBidder(cfg.Adapters[string(openrtb_ext.BidderYieldmo)].Endpoint), openrtb_ext.BidderYieldone: yieldone.NewYieldoneBidder(cfg.Adapters[string(openrtb_ext.BidderYieldone)].Endpoint), openrtb_ext.BidderZeroClickFraud: zeroclickfraud.NewZeroClickFraudBidder(cfg.Adapters[string(openrtb_ext.BidderZeroClickFraud)].Endpoint), + openrtb_ext.BidderBetween: between.NewBetweenBidder(cfg.Adapters[string(openrtb_ext.BidderBetween)].Endpoint), } legacyBidders := map[openrtb_ext.BidderName]adapters.Adapter{ - // TODO #267: Upgrade the Conversant adapter - openrtb_ext.BidderConversant: conversant.NewConversantAdapter(adapters.DefaultHTTPAdapterConfig, cfg.Adapters[string(openrtb_ext.BidderConversant)].Endpoint), // TODO #212: Upgrade the Index adapter openrtb_ext.BidderIx: ix.NewIxAdapter(adapters.DefaultHTTPAdapterConfig, cfg.Adapters[strings.ToLower(string(openrtb_ext.BidderIx))].Endpoint), // TODO #213: Upgrade the Lifestreet adapter @@ -198,7 +228,7 @@ func newAdapterMap(client *http.Client, cfg *config.Configuration, infos adapter for name, bidder := range ortbBidders { // Clean out any disabled bidders if infos[string(name)].Status == adapters.StatusActive { - allBidders[name] = adaptBidder(adapters.EnforceBidderInfo(bidder, infos[string(name)]), client, cfg, me) + allBidders[name] = adaptBidder(adapters.EnforceBidderInfo(bidder, infos[string(name)]), client, cfg, me, name) } } diff --git a/exchange/auction.go b/exchange/auction.go index 1ead3c616c6..646c13bbcb7 100644 --- a/exchange/auction.go +++ b/exchange/auction.go @@ -8,13 +8,13 @@ import ( "fmt" "regexp" "strings" + "time" "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" uuid "github.com/gofrs/uuid" - "github.com/golang/glog" ) type DebugLog struct { @@ -47,7 +47,53 @@ func (d *DebugLog) BuildCacheString() { d.CacheString = fmt.Sprintf("%s%s%s%s", xml.Header, d.Data.Request, d.Data.Headers, d.Data.Response) } -func newAuction(seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, numImps int) *auction { +func (d *DebugLog) PutDebugLogError(cache prebid_cache_client.Client, timeout int, errors []error) error { + if len(d.Data.Response) == 0 && len(errors) == 0 { + d.Data.Response = "No response or errors created" + } + + if len(errors) > 0 { + errStrings := []string{} + for _, err := range errors { + errStrings = append(errStrings, err.Error()) + } + d.Data.Response = fmt.Sprintf("%s\nErrors:\n%s", d.Data.Response, strings.Join(errStrings, "\n")) + } + + d.BuildCacheString() + + if len(d.CacheKey) == 0 { + rawUUID, err := uuid.NewV4() + if err != nil { + return err + } + d.CacheKey = rawUUID.String() + } + + data, err := json.Marshal(d.CacheString) + if err != nil { + return err + } + + toCache := []prebid_cache_client.Cacheable{ + { + Type: d.CacheType, + Data: data, + TTLSeconds: d.TTL, + Key: "log_" + d.CacheKey, + }, + } + + if cache != nil { + ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(time.Duration(timeout)*time.Millisecond)) + defer cancel() + cache.PutJson(ctx, toCache) + } + + return nil +} + +func newAuction(seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, numImps int, preferDeals bool) *auction { winningBids := make(map[string]*pbsOrtbBid, numImps) winningBidsByBidder := make(map[string]map[openrtb_ext.BidderName]*pbsOrtbBid, numImps) @@ -56,7 +102,7 @@ func newAuction(seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, numImps int for _, bid := range seatBid.bids { cpm := bid.bid.Price wbid, ok := winningBids[bid.bid.ImpID] - if !ok || cpm > wbid.bid.Price { + if !ok || isNewWinningBid(bid.bid, wbid.bid, preferDeals) { winningBids[bid.bid.ImpID] = bid } if bidMap, ok := winningBidsByBidder[bid.bid.ImpID]; ok { @@ -78,15 +124,24 @@ func newAuction(seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, numImps int } } +// isNewWinningBid calculates if the new bid (nbid) will win against the current winning bid (wbid) given preferDeals. +func isNewWinningBid(bid, wbid *openrtb.Bid, preferDeals bool) bool { + if preferDeals { + if len(wbid.DealID) > 0 && len(bid.DealID) == 0 { + return false + } + if len(wbid.DealID) == 0 && len(bid.DealID) > 0 { + return true + } + } + return bid.Price > wbid.Price +} + func (a *auction) setRoundedPrices(priceGranularity openrtb_ext.PriceGranularity) { roundedPrices := make(map[*pbsOrtbBid]string, 5*len(a.winningBids)) for _, topBidsPerImp := range a.winningBidsByBidder { for _, topBidPerBidder := range topBidsPerImp { - roundedPrice, err := GetCpmStringValue(topBidPerBidder.bid.Price, priceGranularity) - if err != nil { - glog.Errorf(`Error rounding price according to granularity. This shouldn't happen unless /openrtb2 input validation is buggy. Granularity was "%v".`, priceGranularity) - } - roundedPrices[topBidPerBidder] = roundedPrice + roundedPrices[topBidPerBidder] = GetPriceBucket(topBidPerBidder.bid.Price, priceGranularity) } } a.roundedPrices = roundedPrices @@ -130,7 +185,7 @@ func (a *auction) doCache(ctx context.Context, cache prebid_cache_client.Client, var customCacheKey string var catDur string useCustomCacheKey := false - if competitiveExclusion && isOverallWinner { + if competitiveExclusion && isOverallWinner || includeBidderKeys { // set custom cache key for winning bid when competitive exclusion applies catDur = bidCategory[topBidPerBidder.bid.ID] if len(catDur) > 0 { @@ -179,9 +234,9 @@ func (a *auction) doCache(ctx context.Context, cache prebid_cache_client.Client, } } - if debugLog != nil && debugLog.Enabled { - debugLog.BuildCacheString() + if len(toCache) > 0 && debugLog != nil && debugLog.Enabled { debugLog.CacheKey = hbCacheID + debugLog.BuildCacheString() if jsonBytes, err := json.Marshal(debugLog.CacheString); err == nil { toCache = append(toCache, prebid_cache_client.Cacheable{ Type: debugLog.CacheType, diff --git a/exchange/auction_test.go b/exchange/auction_test.go index 36e06a7d70a..6027a8f4bfd 100644 --- a/exchange/auction_test.go +++ b/exchange/auction_test.go @@ -280,6 +280,225 @@ func runCacheSpec(t *testing.T, fileDisplayName string, specData *cacheSpec) { } } +func TestNewAuction(t *testing.T) { + bid1p077 := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp1", + Price: 0.77, + }, + } + bid1p123 := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp1", + Price: 1.23, + }, + } + bid1p230 := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp1", + Price: 2.30, + }, + } + bid1p088d := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp1", + Price: 0.88, + DealID: "SpecialDeal", + }, + } + bid1p166d := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp1", + Price: 1.66, + DealID: "BigDeal", + }, + } + bid2p123 := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp2", + Price: 1.23, + }, + } + bid2p144 := pbsOrtbBid{ + bid: &openrtb.Bid{ + ImpID: "imp2", + Price: 1.44, + }, + } + tests := []struct { + description string + seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid + numImps int + preferDeals bool + expectedAuction auction + }{ + { + description: "Basic auction test", + seatBids: map[openrtb_ext.BidderName]*pbsOrtbSeatBid{ + "appnexus": { + bids: []*pbsOrtbBid{&bid1p123}, + }, + "rubicon": { + bids: []*pbsOrtbBid{&bid1p230}, + }, + }, + numImps: 1, + preferDeals: false, + expectedAuction: auction{ + winningBids: map[string]*pbsOrtbBid{ + "imp1": &bid1p230, + }, + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "imp1": { + "appnexus": &bid1p123, + "rubicon": &bid1p230, + }, + }, + }, + }, + { + description: "Multi-imp auction", + seatBids: map[openrtb_ext.BidderName]*pbsOrtbSeatBid{ + "appnexus": { + bids: []*pbsOrtbBid{&bid1p230, &bid2p123}, + }, + "rubicon": { + bids: []*pbsOrtbBid{&bid1p077, &bid2p144}, + }, + "openx": { + bids: []*pbsOrtbBid{&bid1p123}, + }, + }, + numImps: 2, + preferDeals: false, + expectedAuction: auction{ + winningBids: map[string]*pbsOrtbBid{ + "imp1": &bid1p230, + "imp2": &bid2p144, + }, + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "imp1": { + "appnexus": &bid1p230, + "rubicon": &bid1p077, + "openx": &bid1p123, + }, + "imp2": { + "appnexus": &bid2p123, + "rubicon": &bid2p144, + }, + }, + }, + }, + { + description: "Basic auction with deals, no preference", + seatBids: map[openrtb_ext.BidderName]*pbsOrtbSeatBid{ + "appnexus": { + bids: []*pbsOrtbBid{&bid1p123}, + }, + "rubicon": { + bids: []*pbsOrtbBid{&bid1p088d}, + }, + }, + numImps: 1, + preferDeals: false, + expectedAuction: auction{ + winningBids: map[string]*pbsOrtbBid{ + "imp1": &bid1p123, + }, + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "imp1": { + "appnexus": &bid1p123, + "rubicon": &bid1p088d, + }, + }, + }, + }, + { + description: "Basic auction with deals, prefer deals", + seatBids: map[openrtb_ext.BidderName]*pbsOrtbSeatBid{ + "appnexus": { + bids: []*pbsOrtbBid{&bid1p123}, + }, + "rubicon": { + bids: []*pbsOrtbBid{&bid1p088d}, + }, + }, + numImps: 1, + preferDeals: true, + expectedAuction: auction{ + winningBids: map[string]*pbsOrtbBid{ + "imp1": &bid1p088d, + }, + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "imp1": { + "appnexus": &bid1p123, + "rubicon": &bid1p088d, + }, + }, + }, + }, + { + description: "Auction with 2 deals", + seatBids: map[openrtb_ext.BidderName]*pbsOrtbSeatBid{ + "appnexus": { + bids: []*pbsOrtbBid{&bid1p166d}, + }, + "rubicon": { + bids: []*pbsOrtbBid{&bid1p088d}, + }, + }, + numImps: 1, + preferDeals: true, + expectedAuction: auction{ + winningBids: map[string]*pbsOrtbBid{ + "imp1": &bid1p166d, + }, + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "imp1": { + "appnexus": &bid1p166d, + "rubicon": &bid1p088d, + }, + }, + }, + }, + { + description: "Auction with 3 bids and 2 deals", + seatBids: map[openrtb_ext.BidderName]*pbsOrtbSeatBid{ + "appnexus": { + bids: []*pbsOrtbBid{&bid1p166d}, + }, + "rubicon": { + bids: []*pbsOrtbBid{&bid1p088d}, + }, + "openx": { + bids: []*pbsOrtbBid{&bid1p230}, + }, + }, + numImps: 1, + preferDeals: true, + expectedAuction: auction{ + winningBids: map[string]*pbsOrtbBid{ + "imp1": &bid1p166d, + }, + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "imp1": { + "appnexus": &bid1p166d, + "rubicon": &bid1p088d, + "openx": &bid1p230, + }, + }, + }, + }, + } + + for _, test := range tests { + auc := newAuction(test.seatBids, test.numImps, test.preferDeals) + + assert.Equal(t, test.expectedAuction, *auc, test.description) + } + +} + type cacheSpec struct { BidRequest openrtb.BidRequest `json:"bidRequest"` PbsBids []pbsBid `json:"pbsBids"` @@ -298,22 +517,27 @@ type pbsBid struct { Bidder openrtb_ext.BidderName `json:"bidder"` } -type mockCache struct { - items []prebid_cache_client.Cacheable -} - type cacheComparator struct { freq int expectedKeys []string actualKeys []string } -func (c *mockCache) GetExtCacheData() (string, string) { - return "", "" +type mockCache struct { + scheme string + host string + path string + items []prebid_cache_client.Cacheable +} + +func (c *mockCache) GetExtCacheData() (scheme string, host string, path string) { + return c.scheme, c.host, c.path } + func (c *mockCache) GetPutUrl() string { return "" } + func (c *mockCache) PutJson(ctx context.Context, values []prebid_cache_client.Cacheable) ([]string, []error) { c.items = values return []string{"", "", "", "", ""}, nil diff --git a/exchange/bidder.go b/exchange/bidder.go index 7e28214890a..e5939f018ed 100644 --- a/exchange/bidder.go +++ b/exchange/bidder.go @@ -8,6 +8,7 @@ import ( "fmt" "io/ioutil" "net/http" + "net/http/httptrace" "time" "github.com/PubMatic-OpenWrap/prebid-server/config/util" @@ -47,7 +48,7 @@ type adaptedBidder interface { // // Any errors will be user-facing in the API. // Error messages should help publishers understand what might account for "bad" bids. - requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (*pbsOrtbSeatBid, []error) + requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (*pbsOrtbSeatBid, []error) } // pbsOrtbBid is a Bid returned by an adaptedBidder. @@ -56,7 +57,8 @@ type adaptedBidder interface { // pbsOrtbBid.bidType will become "response.seatbid[i].bid.ext.prebid.type" in the final OpenRTB response. // pbsOrtbBid.bidTargets does not need to be filled out by the Bidder. It will be set later by the exchange. // pbsOrtbBid.bidVideo is optional but should be filled out by the Bidder if bidType is video. -// pbsOrtbBid.dealPriority will become "response.seatbid[i].bid.dealPriority" in the final OpenRTB response. +// pbsOrtbBid.dealPriority is optionally provided by adapters and used internally by the exchange to support deal targeted campaigns. +// pbsOrtbBid.dealTierSatisfied is set to true by exchange.updateHbPbCatDur if deal tier satisfied otherwise it will be set to false type pbsOrtbBid struct { bid *openrtb.Bid bidType openrtb_ext.BidType @@ -88,23 +90,33 @@ type pbsOrtbSeatBid struct { // // The name refers to the "Adapter" architecture pattern, and should not be confused with a Prebid "Adapter" // (which is being phased out and replaced by Bidder for OpenRTB auctions) -func adaptBidder(bidder adapters.Bidder, client *http.Client, cfg *config.Configuration, me pbsmetrics.MetricsEngine) adaptedBidder { +func adaptBidder(bidder adapters.Bidder, client *http.Client, cfg *config.Configuration, me pbsmetrics.MetricsEngine, name openrtb_ext.BidderName) adaptedBidder { return &bidderAdapter{ - Bidder: bidder, - Client: client, - DebugConfig: cfg.Debug, - me: me, + Bidder: bidder, + BidderName: name, + Client: client, + me: me, + config: bidderAdapterConfig{ + Debug: cfg.Debug, + DisableConnMetrics: cfg.Metrics.Disabled.AdapterConnectionMetrics, + }, } } type bidderAdapter struct { - Bidder adapters.Bidder - Client *http.Client - DebugConfig config.Debug - me pbsmetrics.MetricsEngine + Bidder adapters.Bidder + BidderName openrtb_ext.BidderName + Client *http.Client + me pbsmetrics.MetricsEngine + config bidderAdapterConfig +} + +type bidderAdapterConfig struct { + Debug config.Debug + DisableConnMetrics bool } -func (bidder *bidderAdapter) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (*pbsOrtbSeatBid, []error) { +func (bidder *bidderAdapter) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (*pbsOrtbSeatBid, []error) { reqData, errs := bidder.Bidder.MakeRequests(request, reqInfo) if len(reqData) == 0 { @@ -140,7 +152,7 @@ func (bidder *bidderAdapter) requestBid(ctx context.Context, request *openrtb.Bi for i := 0; i < len(reqData); i++ { httpInfo := <-responseChannel // If this is a test bid, capture debugging info from the requests. - if debug { + if debugInfo := ctx.Value(DebugContextKey); debugInfo != nil && debugInfo.(bool) { seatBid.httpCalls = append(seatBid.httpCalls, makeExt(httpInfo)) } @@ -314,6 +326,10 @@ func makeExt(httpInfo *httpCallInfo) *openrtb_ext.ExtHttpCall { // doRequest makes a request, handles the response, and returns the data needed by the // Bidder interface. func (bidder *bidderAdapter) doRequest(ctx context.Context, req *adapters.RequestData) *httpCallInfo { + return bidder.doRequestImpl(ctx, req, glog.Warningf) +} + +func (bidder *bidderAdapter) doRequestImpl(ctx context.Context, req *adapters.RequestData, logger util.LogMsg) *httpCallInfo { httpReq, err := http.NewRequest(req.Method, req.Uri, bytes.NewBuffer(req.Body)) if err != nil { return &httpCallInfo{ @@ -323,16 +339,27 @@ func (bidder *bidderAdapter) doRequest(ctx context.Context, req *adapters.Reques } httpReq.Header = req.Headers + // If adapter connection metrics are not disabled, add the client trace + // to get complete connection info into our metrics + if !bidder.config.DisableConnMetrics { + ctx = bidder.addClientTrace(ctx) + } httpResp, err := ctxhttp.Do(ctx, bidder.Client, httpReq) if err != nil { if err == context.DeadlineExceeded { err = &errortypes.Timeout{Message: err.Error()} - if tb, ok := bidder.Bidder.(adapters.TimeoutBidder); ok { + var corebidder adapters.Bidder = bidder.Bidder + // The bidder adapter normally stores an info-aware bidder (a bidder wrapper) + // rather than the actual bidder. So we need to unpack that first. + if b, ok := corebidder.(*adapters.InfoAwareBidder); ok { + corebidder = b.Bidder + } + if tb, ok := corebidder.(adapters.TimeoutBidder); ok { // Toss the timeout notification call into a go routine, as we are out of time' // and cannot delay processing. We don't do anything result, as there is not much // we can do about a timeout notification failure. We do not want to get stuck in // a loop of trying to report timeouts to the timeout notifications. - go bidder.doTimeoutNotification(tb, req) + go bidder.doTimeoutNotification(tb, req, logger) } } @@ -368,7 +395,7 @@ func (bidder *bidderAdapter) doRequest(ctx context.Context, req *adapters.Reques } } -func (bidder *bidderAdapter) doTimeoutNotification(timeoutBidder adapters.TimeoutBidder, req *adapters.RequestData) { +func (bidder *bidderAdapter) doTimeoutNotification(timeoutBidder adapters.TimeoutBidder, req *adapters.RequestData, logger util.LogMsg) { ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond) defer cancel() toReq, errL := timeoutBidder.MakeTimeoutNotification(req) @@ -379,7 +406,7 @@ func (bidder *bidderAdapter) doTimeoutNotification(timeoutBidder adapters.Timeou httpResp, err := ctxhttp.Do(ctx, bidder.Client, httpReq) success := (err == nil && httpResp.StatusCode >= 200 && httpResp.StatusCode < 300) bidder.me.RecordTimeoutNotice(success) - if bidder.DebugConfig.TimeoutNotification.Log && !(bidder.DebugConfig.TimeoutNotification.FailOnly && success) { + if bidder.config.Debug.TimeoutNotification.Log && !(bidder.config.Debug.TimeoutNotification.FailOnly && success) { var msg string if err == nil { msg = fmt.Sprintf("TimeoutNotification: status:(%d) body:%s", httpResp.StatusCode, string(toReq.Body)) @@ -387,16 +414,16 @@ func (bidder *bidderAdapter) doTimeoutNotification(timeoutBidder adapters.Timeou msg = fmt.Sprintf("TimeoutNotification: error:(%s) body:%s", err.Error(), string(toReq.Body)) } // If logging is turned on, and logging is not disallowed via FailOnly - util.LogRandomSample(msg, glog.Warningf, bidder.DebugConfig.TimeoutNotification.SamplingRate) + util.LogRandomSample(msg, logger, bidder.config.Debug.TimeoutNotification.SamplingRate) } } else { bidder.me.RecordTimeoutNotice(false) - if bidder.DebugConfig.TimeoutNotification.Log { + if bidder.config.Debug.TimeoutNotification.Log { msg := fmt.Sprintf("TimeoutNotification: Failed to make timeout request: method(%s), uri(%s), error(%s)", toReq.Method, toReq.Uri, err.Error()) - util.LogRandomSample(msg, glog.Warningf, bidder.DebugConfig.TimeoutNotification.SamplingRate) + util.LogRandomSample(msg, logger, bidder.config.Debug.TimeoutNotification.SamplingRate) } } - } else if bidder.DebugConfig.TimeoutNotification.Log { + } else if bidder.config.Debug.TimeoutNotification.Log { reqJSON, err := json.Marshal(req) var msg string if err == nil { @@ -404,7 +431,7 @@ func (bidder *bidderAdapter) doTimeoutNotification(timeoutBidder adapters.Timeou } else { msg = fmt.Sprintf("TimeoutNotification: Failed to generate timeout request: error(%s), bidder request marshal failed(%s)", errL[0].Error(), err.Error()) } - util.LogRandomSample(msg, glog.Warningf, bidder.DebugConfig.TimeoutNotification.SamplingRate) + util.LogRandomSample(msg, logger, bidder.config.Debug.TimeoutNotification.SamplingRate) } } @@ -414,3 +441,34 @@ type httpCallInfo struct { response *adapters.ResponseData err error } + +// This function adds an httptrace.ClientTrace object to the context so, if connection with the bidder +// endpoint is established, we can keep track of whether the connection was newly created, reused, and +// the time from the connection request, to the connection creation. +func (bidder *bidderAdapter) addClientTrace(ctx context.Context) context.Context { + var connStart, dnsStart time.Time + + trace := &httptrace.ClientTrace{ + // GetConn is called before a connection is created or retrieved from an idle pool + GetConn: func(hostPort string) { + connStart = time.Now() + }, + // GotConn is called after a successful connection is obtained + GotConn: func(info httptrace.GotConnInfo) { + connWaitTime := time.Now().Sub(connStart) + + bidder.me.RecordAdapterConnections(bidder.BidderName, info.Reused, connWaitTime) + }, + // DNSStart is called when a DNS lookup begins. + DNSStart: func(info httptrace.DNSStartInfo) { + dnsStart = time.Now() + }, + // DNSDone is called when a DNS lookup ends. + DNSDone: func(info httptrace.DNSDoneInfo) { + dnsLookupTime := time.Now().Sub(dnsStart) + + bidder.me.RecordDNSTime(dnsLookupTime) + }, + } + return httptrace.WithClientTrace(ctx, trace) +} diff --git a/exchange/bidder_test.go b/exchange/bidder_test.go index ebf9eccbf9d..9e27bc41477 100644 --- a/exchange/bidder_test.go +++ b/exchange/bidder_test.go @@ -1,12 +1,16 @@ package exchange import ( + "bytes" "context" "encoding/json" "errors" "fmt" + "io/ioutil" "net/http" "net/http/httptest" + "net/http/httptrace" + "strings" "testing" "time" @@ -15,8 +19,12 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/currencies" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + metricsConf "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics/config" metricsConfig "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics/config" + "github.com/golang/glog" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" nativeRequests "github.com/PubMatic-OpenWrap/openrtb/native/request" nativeResponse "github.com/PubMatic-OpenWrap/openrtb/native/response" @@ -66,9 +74,9 @@ func TestSingleBidder(t *testing.T) { }, bidResponse: mockBidderResponse, } - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) - currencyConverter := currencies.NewRateConverterDefault() - seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", bidAdjustment, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", bidAdjustment, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) // Make sure the goodSingleBidder was called with the expected arguments. if bidderImpl.httpResponse == nil { @@ -154,9 +162,9 @@ func TestMultiBidder(t *testing.T) { }}, bidResponse: mockBidderResponse, } - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) - currencyConverter := currencies.NewRateConverterDefault() - seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if seatBid == nil { t.Fatalf("SeatBid should exist, because bids exist.") @@ -192,8 +200,10 @@ func TestBidderTimeout(t *testing.T) { defer server.Close() bidder := &bidderAdapter{ - Bidder: &mixedMultiBidder{}, - Client: server.Client(), + Bidder: &mixedMultiBidder{}, + BidderName: openrtb_ext.BidderAppnexus, + Client: server.Client(), + me: &metricsConf.DummyMetricsEngine{}, } callInfo := bidder.doRequest(ctx, &adapters.RequestData{ @@ -233,8 +243,10 @@ func TestConnectionClose(t *testing.T) { server = httptest.NewServer(handler) bidder := &bidderAdapter{ - Bidder: &mixedMultiBidder{}, - Client: server.Client(), + Bidder: &mixedMultiBidder{}, + Client: server.Client(), + BidderName: openrtb_ext.BidderAppnexus, + me: &metricsConf.DummyMetricsEngine{}, } callInfo := bidder.doRequest(context.Background(), &adapters.RequestData{ @@ -512,12 +524,15 @@ func TestMultiCurrencies(t *testing.T) { ) // Execute: - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) currencyConverter := currencies.NewRateConverter( &http.Client{}, mockedHTTPServer.URL, - time.Duration(10)*time.Second, + time.Duration(24)*time.Hour, ) + time.Sleep(time.Duration(500) * time.Millisecond) + currencyConverter.Run() + seatBid, errs := bidder.requestBid( context.Background(), &openrtb.BidRequest{}, @@ -525,7 +540,6 @@ func TestMultiCurrencies(t *testing.T) { 1, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, - false, ) // Verify: @@ -661,8 +675,8 @@ func TestMultiCurrencies_RateConverterNotSet(t *testing.T) { } // Execute: - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) - currencyConverter := currencies.NewRateConverterDefault() + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) seatBid, errs := bidder.requestBid( context.Background(), &openrtb.BidRequest{}, @@ -670,7 +684,6 @@ func TestMultiCurrencies_RateConverterNotSet(t *testing.T) { 1, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, - false, ) // Verify: @@ -828,11 +841,11 @@ func TestMultiCurrencies_RequestCurrencyPick(t *testing.T) { } // Execute: - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) currencyConverter := currencies.NewRateConverter( &http.Client{}, mockedHTTPServer.URL, - time.Duration(10)*time.Second, + time.Duration(24)*time.Hour, ) seatBid, errs := bidder.requestBid( context.Background(), @@ -843,7 +856,6 @@ func TestMultiCurrencies_RequestCurrencyPick(t *testing.T) { 1, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, - false, ) // Verify: @@ -927,55 +939,6 @@ func TestSuccessfulResponseLogging(t *testing.T) { } } -// TestServerCallDebugging makes sure that we log the server calls made by the Bidder on test bids. -func TestServerCallDebugging(t *testing.T) { - respBody := "{\"bid\":false}" - respStatus := 200 - server := httptest.NewServer(mockHandler(respStatus, "getBody", respBody)) - defer server.Close() - - reqBody := "{\"key\":\"val\"}" - reqUrl := server.URL - bidderImpl := &goodSingleBidder{ - httpRequest: &adapters.RequestData{ - Method: "POST", - Uri: reqUrl, - Body: []byte(reqBody), - Headers: http.Header{}, - }, - } - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) - currencyConverter := currencies.NewRateConverterDefault() - - bids, _ := bidder.requestBid( - context.Background(), - &openrtb.BidRequest{ - Test: 1, - }, - "test", - 1.0, - currencyConverter.Rates(), - &adapters.ExtraRequestInfo{}, - true, - ) - - if len(bids.httpCalls) != 1 { - t.Errorf("We should log the server call if this is a test bid. Got %d", len(bids.httpCalls)) - } - if bids.httpCalls[0].Uri != reqUrl { - t.Errorf("Wrong httpcalls URI. Expected %s, got %s", reqUrl, bids.httpCalls[0].Uri) - } - if bids.httpCalls[0].RequestBody != reqBody { - t.Errorf("Wrong httpcalls RequestBody. Expected %s, got %s", reqBody, bids.httpCalls[0].RequestBody) - } - if bids.httpCalls[0].ResponseBody != respBody { - t.Errorf("Wrong httpcalls ResponseBody. Expected %s, got %s", respBody, bids.httpCalls[0].ResponseBody) - } - if bids.httpCalls[0].Status != respStatus { - t.Errorf("Wrong httpcalls Status. Expected %d, got %d", respStatus, bids.httpCalls[0].Status) - } -} - func TestMobileNativeTypes(t *testing.T) { respBody := "{\"bid\":false}" respStatus := 200 @@ -1057,8 +1020,8 @@ func TestMobileNativeTypes(t *testing.T) { }, bidResponse: tc.mockBidderResponse, } - bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) - currencyConverter := currencies.NewRateConverterDefault() + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) seatBids, _ := bidder.requestBid( context.Background(), @@ -1067,7 +1030,6 @@ func TestMobileNativeTypes(t *testing.T) { 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, - false, ) var actualValue string @@ -1079,9 +1041,9 @@ func TestMobileNativeTypes(t *testing.T) { } func TestErrorReporting(t *testing.T) { - bidder := adaptBidder(&bidRejector{}, nil, &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) - currencyConverter := currencies.NewRateConverterDefault() - bids, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + bidder := adaptBidder(&bidRejector{}, nil, &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + bids, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if bids != nil { t.Errorf("There should be no seatbid if no http requests are returned.") } @@ -1234,14 +1196,91 @@ func TestSetAssetTypes(t *testing.T) { } } +func TestCallRecordAdapterConnections(t *testing.T) { + // Setup mock server + respStatus := 200 + respBody := "{\"bid\":false}" + server := httptest.NewServer(mockHandler(respStatus, "getBody", respBody)) + defer server.Close() + + // declare requestBid parameters + bidAdjustment := 2.0 + + bidderImpl := &goodSingleBidder{ + httpRequest: &adapters.RequestData{ + Method: "POST", + Uri: server.URL, + Body: []byte("{\"key\":\"val\"}"), + Headers: http.Header{}, + }, + bidResponse: &adapters.BidderResponse{}, + } + + // setup a mock metrics engine and its expectation + metrics := &pbsmetrics.MetricsEngineMock{} + expectedAdapterName := openrtb_ext.BidderAppnexus + compareConnWaitTime := func(dur time.Duration) bool { return dur.Nanoseconds() > 0 } + + metrics.On("RecordAdapterConnections", expectedAdapterName, false, mock.MatchedBy(compareConnWaitTime)).Once() + + // Run requestBid using an http.Client with a mock handler + bidder := adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, metrics, openrtb_ext.BidderAppnexus) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + _, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, "test", bidAdjustment, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) + + // Assert no errors + assert.Equal(t, 0, len(errs), "bidder.requestBid returned errors %v \n", errs) + + // Assert RecordAdapterConnections() was called with the parameters we expected + metrics.AssertExpectations(t) +} + +type DNSDoneTripper struct{} + +func (DNSDoneTripper) RoundTrip(req *http.Request) (*http.Response, error) { + //Access the httptrace.ClientTrace + trace := httptrace.ContextClientTrace(req.Context()) + + //Force DNSDone call defined in exchange/bidder.go + trace.DNSDone(httptrace.DNSDoneInfo{}) + + resp := &http.Response{ + StatusCode: 200, + Header: map[string][]string{"Location": {"http://www.example.com/"}}, + Body: ioutil.NopCloser(strings.NewReader("postBody")), + } + return resp, nil +} + +func TestCallRecordRecordDNSTime(t *testing.T) { + // setup a mock metrics engine and its expectation + metricsMock := &pbsmetrics.MetricsEngineMock{} + metricsMock.Mock.On("RecordDNSTime", mock.Anything).Return() + + // Instantiate the bidder that will send the request. We'll make sure to use an + // http.Client that runs our mock RoundTripper so DNSDone(httptrace.DNSDoneInfo{}) + // gets called + bidder := &bidderAdapter{ + Bidder: &mixedMultiBidder{}, + Client: &http.Client{Transport: DNSDoneTripper{}}, + me: metricsMock, + } + + // Run test + bidder.doRequest(context.Background(), &adapters.RequestData{Method: "POST", Uri: "http://www.example.com/"}) + + // Tried one or another, none seem to work without panicking + metricsMock.AssertExpectations(t) +} + func TestTimeoutNotificationOff(t *testing.T) { respBody := "{\"bid\":false}" respStatus := 200 server := httptest.NewServer(mockHandler(respStatus, "getBody", respBody)) defer server.Close() - bidderImpl := ¬ifingBidder{ - notiRequest: adapters.RequestData{ + bidderImpl := ¬ifyingBidder{ + notifyRequest: adapters.RequestData{ Method: "GET", Uri: server.URL + "/notify/me", Body: nil, @@ -1249,47 +1288,93 @@ func TestTimeoutNotificationOff(t *testing.T) { }, } bidder := &bidderAdapter{ - Bidder: bidderImpl, - Client: server.Client(), - DebugConfig: config.Debug{}, - me: &metricsConfig.DummyMetricsEngine{}, + Bidder: bidderImpl, + Client: server.Client(), + config: bidderAdapterConfig{Debug: config.Debug{}}, + me: &metricsConf.DummyMetricsEngine{}, } if tb, ok := bidder.Bidder.(adapters.TimeoutBidder); !ok { t.Error("Failed to cast bidder to a TimeoutBidder") } else { - bidder.doTimeoutNotification(tb, &adapters.RequestData{}) + bidder.doTimeoutNotification(tb, &adapters.RequestData{}, glog.Warningf) } } func TestTimeoutNotificationOn(t *testing.T) { - respBody := "{\"bid\":false}" - respStatus := 200 - server := httptest.NewServer(mockHandler(respStatus, "getBody", respBody)) + // Expire context immediately to force timeout handler. + ctx, cancelFunc := context.WithDeadline(context.Background(), time.Now()) + cancelFunc() + + // Notification logic is hardcoded for 200ms. We need to wait for a little longer than that. + server := httptest.NewServer(mockSlowHandler(205*time.Millisecond, 200, `{"bid":false}`)) defer server.Close() - bidderImpl := ¬ifingBidder{ - notiRequest: adapters.RequestData{ + bidder := ¬ifyingBidder{ + notifyRequest: adapters.RequestData{ Method: "GET", Uri: server.URL + "/notify/me", Body: nil, Headers: http.Header{}, }, } - bidder := &bidderAdapter{ - Bidder: bidderImpl, + + // Wrap with BidderInfo to mimic exchange.go flow. + bidderWrappedWithInfo := wrapWithBidderInfo(bidder) + + bidderAdapter := &bidderAdapter{ + Bidder: bidderWrappedWithInfo, Client: server.Client(), - DebugConfig: config.Debug{ - TimeoutNotification: config.TimeoutNotification{ - Log: true, + config: bidderAdapterConfig{ + Debug: config.Debug{ + TimeoutNotification: config.TimeoutNotification{ + Log: true, + SamplingRate: 1.0, + }, }, }, - me: &metricsConfig.DummyMetricsEngine{}, + me: &metricsConf.DummyMetricsEngine{}, } - if tb, ok := bidder.Bidder.(adapters.TimeoutBidder); !ok { - t.Error("Failed to cast bidder to a TimeoutBidder") - } else { - bidder.doTimeoutNotification(tb, &adapters.RequestData{}) + + // Unwrap To Mimic exchange.go Casting Code + var coreBidder adapters.Bidder = bidderAdapter.Bidder + if b, ok := coreBidder.(*adapters.InfoAwareBidder); ok { + coreBidder = b.Bidder + } + if _, ok := coreBidder.(adapters.TimeoutBidder); !ok { + t.Fatal("Failed to cast bidder to a TimeoutBidder") + } + + bidRequest := adapters.RequestData{ + Method: "POST", + Uri: server.URL, + Body: []byte(`{"id":"this-id","app":{"publisher":{"id":"pub-id"}}}`), } + + var loggerBuffer bytes.Buffer + logger := func(msg string, args ...interface{}) { + loggerBuffer.WriteString(fmt.Sprintf(fmt.Sprintln(msg), args...)) + } + + bidderAdapter.doRequestImpl(ctx, &bidRequest, logger) + + // Wait a little longer than the 205ms mock server sleep. + time.Sleep(210 * time.Millisecond) + + logExpected := "TimeoutNotification: error:(context deadline exceeded) body:\n" + logActual := loggerBuffer.String() + assert.EqualValues(t, logExpected, logActual) +} + +func wrapWithBidderInfo(bidder adapters.Bidder) adapters.Bidder { + bidderInfo := adapters.BidderInfo{ + Status: adapters.StatusActive, + Capabilities: &adapters.CapabilitiesInfo{ + App: &adapters.PlatformInfo{ + MediaTypes: []openrtb_ext.BidType{openrtb_ext.BidTypeBanner}, + }, + }, + } + return adapters.EnforceBidderInfo(bidder, bidderInfo) } type goodSingleBidder struct { @@ -1366,18 +1451,19 @@ func (bidder *bidRejector) MakeBids(internalRequest *openrtb.BidRequest, externa return nil, []error{errors.New("Can't make a response.")} } -type notifingBidder struct { - notiRequest adapters.RequestData +type notifyingBidder struct { + requests []*adapters.RequestData + notifyRequest adapters.RequestData } -func (bidder *notifingBidder) MakeRequests(request *openrtb.BidRequest, reqInfo *adapters.ExtraRequestInfo) ([]*adapters.RequestData, []error) { - return nil, nil +func (bidder *notifyingBidder) MakeRequests(request *openrtb.BidRequest, reqInfo *adapters.ExtraRequestInfo) ([]*adapters.RequestData, []error) { + return bidder.requests, nil } -func (bidder *notifingBidder) MakeBids(internalRequest *openrtb.BidRequest, externalRequest *adapters.RequestData, response *adapters.ResponseData) (*adapters.BidderResponse, []error) { +func (bidder *notifyingBidder) MakeBids(internalRequest *openrtb.BidRequest, externalRequest *adapters.RequestData, response *adapters.ResponseData) (*adapters.BidderResponse, []error) { return nil, nil } -func (bidder *notifingBidder) MakeTimeoutNotification(req *adapters.RequestData) (*adapters.RequestData, []error) { - return &bidder.notiRequest, nil +func (bidder *notifyingBidder) MakeTimeoutNotification(req *adapters.RequestData) (*adapters.RequestData, []error) { + return &bidder.notifyRequest, nil } diff --git a/exchange/bidder_validate_bids.go b/exchange/bidder_validate_bids.go index 723515800ba..9981a83f54c 100644 --- a/exchange/bidder_validate_bids.go +++ b/exchange/bidder_validate_bids.go @@ -7,11 +7,9 @@ import ( "strings" "github.com/PubMatic-OpenWrap/openrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters" "github.com/PubMatic-OpenWrap/prebid-server/currencies" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" - - "github.com/PubMatic-OpenWrap/prebid-server/adapters" - "golang.org/x/text/currency" ) @@ -30,8 +28,8 @@ type validatedBidder struct { bidder adaptedBidder } -func (v *validatedBidder) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (*pbsOrtbSeatBid, []error) { - seatBid, errs := v.bidder.requestBid(ctx, request, name, bidAdjustment, conversions, reqInfo, debug) +func (v *validatedBidder) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (*pbsOrtbSeatBid, []error) { + seatBid, errs := v.bidder.requestBid(ctx, request, name, bidAdjustment, conversions, reqInfo) if validationErrors := removeInvalidBids(request, seatBid); len(validationErrors) > 0 { errs = append(errs, validationErrors...) } diff --git a/exchange/bidder_validate_bids_test.go b/exchange/bidder_validate_bids_test.go index 332a67d8c62..99f687ff6bc 100644 --- a/exchange/bidder_validate_bids_test.go +++ b/exchange/bidder_validate_bids_test.go @@ -42,7 +42,7 @@ func TestAllValidBids(t *testing.T) { }, }, }) - seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}, false) + seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}) assert.Len(t, seatBid.bids, 3) assert.Len(t, errs, 0) } @@ -83,7 +83,7 @@ func TestAllBadBids(t *testing.T) { }, }, }) - seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}, false) + seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}) assert.Len(t, seatBid.bids, 0) assert.Len(t, errs, 5) } @@ -126,7 +126,7 @@ func TestMixedBids(t *testing.T) { }, }, }) - seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}, false) + seatBid, errs := bidder.requestBid(context.Background(), &openrtb.BidRequest{}, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}) assert.Len(t, seatBid.bids, 2) assert.Len(t, errs, 3) } @@ -246,7 +246,7 @@ func TestCurrencyBids(t *testing.T) { Cur: tc.brqCur, } - seatBid, errs := bidder.requestBid(context.Background(), request, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}, false) + seatBid, errs := bidder.requestBid(context.Background(), request, openrtb_ext.BidderAppnexus, 1.0, currencies.NewConstantRates(), &adapters.ExtraRequestInfo{}) assert.Len(t, seatBid.bids, expectedValidBids) assert.Len(t, errs, expectedErrs) } @@ -257,6 +257,6 @@ type mockAdaptedBidder struct { errorResponse []error } -func (b *mockAdaptedBidder) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (*pbsOrtbSeatBid, []error) { +func (b *mockAdaptedBidder) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (*pbsOrtbSeatBid, []error) { return b.bidResponse, b.errorResponse } diff --git a/exchange/cachetest/customcachekey.json b/exchange/cachetest/customcachekey.json index bb2a37ca356..9a9008fe5d7 100644 --- a/exchange/cachetest/customcachekey.json +++ b/exchange/cachetest/customcachekey.json @@ -26,14 +26,14 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"appbid001\",\"impid\": \"oneImp\",\"price\": 7.64,\"cat\": [\"11_sports_22\"]}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"appbid001\",\"impid\": \"oneImp\",\"price\": 7.64,\"cat\": [\"11_sports_22\"]}" }, { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"pubbid001\", \"impid\": \"oneImp\", \"price\": 5.64, \"cat\": [\"33_news_44\"]}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"pubbid001\", \"impid\": \"oneImp\", \"price\": 5.64, \"cat\": [\"33_news_44\"]}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/customcachekey_no_bidders.json b/exchange/cachetest/customcachekey_no_bidders.json index b8521582a47..9824ed55d15 100644 --- a/exchange/cachetest/customcachekey_no_bidders.json +++ b/exchange/cachetest/customcachekey_no_bidders.json @@ -26,9 +26,9 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"appbid001\", \"impid\": \"oneImp\", \"price\": 7.64, \"cat\": [\"11_sports_22\"]}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"appbid001\", \"impid\": \"oneImp\", \"price\": 7.64, \"cat\": [\"11_sports_22\"]}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/customcachekey_no_winners.json b/exchange/cachetest/customcachekey_no_winners.json index f6204db37f5..75ef628e175 100644 --- a/exchange/cachetest/customcachekey_no_winners.json +++ b/exchange/cachetest/customcachekey_no_winners.json @@ -26,14 +26,14 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"appbid001\", \"impid\": \"oneImp\", \"price\": 7.64, \"cat\": [\"11_sports_22\"]}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"appbid001\", \"impid\": \"oneImp\", \"price\": 7.64, \"cat\": [\"11_sports_22\"]}" }, { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\":\"pubbid001\",\"impid\":\"oneImp\",\"price\":5.64,\"cat\":[\"33_news_44\"]}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\":\"pubbid001\",\"impid\":\"oneImp\",\"price\":5.64,\"cat\":[\"33_news_44\"]}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/debuglog_disabled.json b/exchange/cachetest/debuglog_disabled.json index 88d6332cb09..a7efeff0db5 100644 --- a/exchange/cachetest/debuglog_disabled.json +++ b/exchange/cachetest/debuglog_disabled.json @@ -36,13 +36,13 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" }, { - "Type": "json", - "TTLSeconds": 3660, - "Data": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64}" + "type": "json", + "ttlseconds": 3660, + "value": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/debuglog_enabled.json b/exchange/cachetest/debuglog_enabled.json index 670b694f7a7..e6c85c57055 100644 --- a/exchange/cachetest/debuglog_enabled.json +++ b/exchange/cachetest/debuglog_enabled.json @@ -36,17 +36,17 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" }, { - "Type": "json", - "TTLSeconds": 3660, - "Data": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64}" + "type": "json", + "ttlseconds": 3660, + "value": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64}" }, { - "Type": "xml", - "TTLSeconds": 3600, - "Data": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cLog\u003e\u003cRequest\u003etest request string\u003c/Request\u003e\u003cHeaders\u003etest headers string\u003c/Headers\u003e\u003cResponse\u003etest response string\u003c/Response\u003e\u003c/Log\u003e" + "type": "xml", + "ttlseconds": 3600, + "value": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cLog\u003e\u003cRequest\u003etest request string\u003c/Request\u003e\u003cHeaders\u003etest headers string\u003c/Headers\u003e\u003cResponse\u003etest response string\u003c/Response\u003e\u003c/Log\u003e" } ], "defaultTTLs": { diff --git a/exchange/cachetest/debuglog_enabled_no_winners_nor_bids.json b/exchange/cachetest/debuglog_enabled_no_winners_nor_bids.json new file mode 100644 index 00000000000..637b33e171b --- /dev/null +++ b/exchange/cachetest/debuglog_enabled_no_winners_nor_bids.json @@ -0,0 +1,54 @@ +{ + "debugLog": { + "Enabled": true, + "CacheType": "xml", + "TTL": 3600, + "Data": { + "Request": "test request string", + "Headers": "test headers string", + "Response": "test response string" + } + }, + "bidRequest": { + "imp": [ + { + "id": "oneImp", + "exp": 600 + }, + { + "id": "twoImp" + } + ] + }, + "pbsBids": [ + { + "bid": { + "id": "bidOne", + "impid": "oneImp", + "price": 7.64 + }, + "bidType": "video", + "bidder": "appnexus" + }, + { + "bid": { + "id": "bidTwo", + "impid": "twoImp", + "price": 5.64 + }, + "bidType": "video", + "bidder": "pubmatic" + } + ], + "expectedCacheables": [], + "defaultTTLs": { + "banner": 300, + "video": 3600, + "audio": 1800, + "native": 300 + }, + "targetDataIncludeWinners": false, + "targetDataIncludeBidderKeys": false, + "targetDataIncludeCacheBids": true, + "targetDataIncludeCacheVast": false +} \ No newline at end of file diff --git a/exchange/cachetest/defaultbanner.json b/exchange/cachetest/defaultbanner.json index ca44589cb1b..8bc59f632fe 100644 --- a/exchange/cachetest/defaultbanner.json +++ b/exchange/cachetest/defaultbanner.json @@ -24,13 +24,13 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600}" + "type": "json", + "ttlseconds": 660, + "value":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600}" }, { - "Type": "json", - "TTLSeconds": 360, - "Data": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64 }" + "type": "json", + "ttlseconds": 360, + "value": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64 }" } ], "defaultTTLs": { diff --git a/exchange/cachetest/defaultbanner_no_bidders.json b/exchange/cachetest/defaultbanner_no_bidders.json index d517182168d..6dc534b5c6f 100644 --- a/exchange/cachetest/defaultbanner_no_bidders.json +++ b/exchange/cachetest/defaultbanner_no_bidders.json @@ -24,9 +24,9 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" + "type": "json", + "ttlseconds": 660, + "value":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" } ], "defaultTTLs": { diff --git a/exchange/cachetest/defaultbanner_no_winners.json b/exchange/cachetest/defaultbanner_no_winners.json index fe43462b241..154e1faa600 100644 --- a/exchange/cachetest/defaultbanner_no_winners.json +++ b/exchange/cachetest/defaultbanner_no_winners.json @@ -24,13 +24,13 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" + "type": "json", + "ttlseconds": 660, + "value":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" }, { - "Type": "json", - "TTLSeconds": 360, - "Data": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64 }" + "type": "json", + "ttlseconds": 360, + "value": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64 }" } ], "defaultTTLs": { diff --git a/exchange/cachetest/defaultvideo.json b/exchange/cachetest/defaultvideo.json index 4d38585ddf1..8d7fcfed836 100644 --- a/exchange/cachetest/defaultvideo.json +++ b/exchange/cachetest/defaultvideo.json @@ -26,13 +26,13 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" }, { - "Type": "json", - "TTLSeconds": 3660, - "Data": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64}" + "type": "json", + "ttlseconds": 3660, + "value": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/defaultvideo_no_bidders.json b/exchange/cachetest/defaultvideo_no_bidders.json index 0a9e9f1de61..ab8f13ff5d5 100644 --- a/exchange/cachetest/defaultvideo_no_bidders.json +++ b/exchange/cachetest/defaultvideo_no_bidders.json @@ -26,9 +26,9 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/defaultvideo_no_winners.json b/exchange/cachetest/defaultvideo_no_winners.json index 98a12a5ad2b..21ad558ce4c 100644 --- a/exchange/cachetest/defaultvideo_no_winners.json +++ b/exchange/cachetest/defaultvideo_no_winners.json @@ -28,13 +28,13 @@ }], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 660, - "Data": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"nurl\": \"http://domain.com/win-notify/1\"}" + "type": "json", + "ttlseconds": 660, + "value": "{\"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"nurl\": \"http://domain.com/win-notify/1\"}" }, { - "Type": "json", - "TTLSeconds": 3660, - "Data": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64, \"nurl\": \"http://domain.com/win-notify/1\"}" + "type": "json", + "ttlseconds": 3660, + "value": "{\"id\": \"bidTwo\", \"impid\": \"twoImp\", \"price\": 5.64, \"nurl\": \"http://domain.com/win-notify/1\"}" } ], "defaultTTLs": { diff --git a/exchange/cachetest/multibid.json b/exchange/cachetest/multibid.json index f2405466235..09095bd51f2 100644 --- a/exchange/cachetest/multibid.json +++ b/exchange/cachetest/multibid.json @@ -54,25 +54,25 @@ ], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 360, - "Data":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" + "type": "json", + "ttlseconds": 360, + "value":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" }, { - "Type": "json", - "TTLSeconds": 260, - "Data": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64, \"exp\": 200 }" + "type": "json", + "ttlseconds": 260, + "value": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64, \"exp\": 200 }" }, { - "Type": "json", - "TTLSeconds": 360, - "Data": "{ \"id\": \"bidThree\", \"impid\": \"oneImp\", \"price\": 2.3 }" + "type": "json", + "ttlseconds": 360, + "value": "{ \"id\": \"bidThree\", \"impid\": \"oneImp\", \"price\": 2.3 }" }, { - "Type": "json", - "TTLSeconds": 0, - "Data": "{ \"id\": \"bidFour\", \"impid\": \"twoImp\", \"price\": 1.64 }" + "type": "json", + "ttlseconds": 0, + "value": "{ \"id\": \"bidFour\", \"impid\": \"twoImp\", \"price\": 1.64 }" }, { - "Type": "json", - "TTLSeconds": 960, - "Data": "{ \"id\": \"bidFive\", \"impid\": \"twoImp\", \"price\": 7.64, \"exp\": 900 }" + "type": "json", + "ttlseconds": 960, + "value": "{ \"id\": \"bidFive\", \"impid\": \"twoImp\", \"price\": 7.64, \"exp\": 900 }" } ], "targetDataIncludeWinners":true, diff --git a/exchange/cachetest/multibid_no_bidders.json b/exchange/cachetest/multibid_no_bidders.json index 1ec47579daf..446ae9ca189 100644 --- a/exchange/cachetest/multibid_no_bidders.json +++ b/exchange/cachetest/multibid_no_bidders.json @@ -54,13 +54,13 @@ ], "expectedCacheables": [ { - "Type": "json", - "Data": "{\"id\": \"bidOne\",\"impid\": \"oneImp\",\"price\": 7.64,\"exp\": 600}", - "TTLSeconds": 360 + "type": "json", + "value": "{\"id\": \"bidOne\",\"impid\": \"oneImp\",\"price\": 7.64,\"exp\": 600}", + "ttlseconds": 360 }, { - "Type": "json", - "Data": "{\"id\": \"bidFive\", \"impid\": \"twoImp\", \"price\": 7.64, \"exp\": 900}", - "TTLSeconds": 960 + "type": "json", + "value": "{\"id\": \"bidFive\", \"impid\": \"twoImp\", \"price\": 7.64, \"exp\": 900}", + "ttlseconds": 960 } ], "targetDataIncludeWinners":true, diff --git a/exchange/cachetest/multibid_no_winners.json b/exchange/cachetest/multibid_no_winners.json index 2221a54ca3c..2f260076d18 100644 --- a/exchange/cachetest/multibid_no_winners.json +++ b/exchange/cachetest/multibid_no_winners.json @@ -54,25 +54,25 @@ ], "expectedCacheables": [ { - "Type": "json", - "TTLSeconds": 360, - "Data":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" + "type": "json", + "ttlseconds": 360, + "value":"{ \"id\": \"bidOne\", \"impid\": \"oneImp\", \"price\": 7.64, \"exp\": 600 }" }, { - "Type": "json", - "TTLSeconds": 260, - "Data": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64, \"exp\": 200 }" + "type": "json", + "ttlseconds": 260, + "value": "{ \"id\": \"bidTwo\", \"impid\": \"oneImp\", \"price\": 5.64, \"exp\": 200 }" }, { - "Type": "json", - "TTLSeconds": 360, - "Data": "{ \"id\": \"bidThree\", \"impid\": \"oneImp\", \"price\": 2.3 }" + "type": "json", + "ttlseconds": 360, + "value": "{ \"id\": \"bidThree\", \"impid\": \"oneImp\", \"price\": 2.3 }" }, { - "Type": "json", - "TTLSeconds": 0, - "Data": "{ \"id\": \"bidFour\", \"impid\": \"twoImp\", \"price\": 1.64 }" + "type": "json", + "ttlseconds": 0, + "value": "{ \"id\": \"bidFour\", \"impid\": \"twoImp\", \"price\": 1.64 }" }, { - "Type": "json", - "TTLSeconds": 960, - "Data": "{ \"id\": \"bidFive\", \"impid\": \"twoImp\", \"price\": 7.64, \"exp\": 900 }" + "type": "json", + "ttlseconds": 960, + "value": "{ \"id\": \"bidFive\", \"impid\": \"twoImp\", \"price\": 7.64, \"exp\": 900 }" } ], "targetDataIncludeWinners":false, diff --git a/exchange/customcachekeytest/customcachekey.json b/exchange/customcachekeytest/customcachekey.json index e578f980943..9b903575edc 100644 --- a/exchange/customcachekeytest/customcachekey.json +++ b/exchange/customcachekeytest/customcachekey.json @@ -28,14 +28,14 @@ }], "expectedCacheables": [ { - "Type": "xml", - "TTLSeconds": 660, - "Key": "11_sports_22_", - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 660, + "key": "11_sports_22_", + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" }, { - "Type": "xml", - "TTLSeconds": 660, - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://anotherdomain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 660, + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://anotherdomain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" } ], "defaultTTLs": { diff --git a/exchange/customcachekeytest/customcachekey_no_bidders.json b/exchange/customcachekeytest/customcachekey_no_bidders.json index 0f09c8dbb9d..589d1e6b4f1 100644 --- a/exchange/customcachekeytest/customcachekey_no_bidders.json +++ b/exchange/customcachekeytest/customcachekey_no_bidders.json @@ -27,10 +27,10 @@ }], "expectedCacheables": [ { - "Type": "xml", - "TTLSeconds": 660, - "Key": "11_sports_22_", - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 660, + "key": "11_sports_22_", + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" } ], "defaultTTLs": { diff --git a/exchange/customcachekeytest/customcachekey_no_winners.json b/exchange/customcachekeytest/customcachekey_no_winners.json index f21c8bda6a1..3eaf6cfb46a 100644 --- a/exchange/customcachekeytest/customcachekey_no_winners.json +++ b/exchange/customcachekeytest/customcachekey_no_winners.json @@ -27,14 +27,14 @@ }], "expectedCacheables": [ { - "Type": "xml", - "TTLSeconds": 660, - "Key": "11_sports_22_", - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 660, + "key": "11_sports_22_", + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" }, { - "Type": "xml", - "TTLSeconds": 660, - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 660, + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" } ], "defaultTTLs": { diff --git a/exchange/exchange.go b/exchange/exchange.go index a898e608355..8fad1d748ad 100644 --- a/exchange/exchange.go +++ b/exchange/exchange.go @@ -7,15 +7,16 @@ import ( "errors" "fmt" "math/rand" - - // "math/rand" "net/http" + "net/url" "runtime/debug" "sort" + "strconv" "strings" "time" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + uuid "github.com/gofrs/uuid" "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/adapters" @@ -29,16 +30,25 @@ import ( "github.com/golang/glog" ) +type ContextKey string + +const DebugContextKey = ContextKey("debugInfo") + +type extCacheInstructions struct { + cacheBids, cacheVAST, returnCreative bool +} + // Exchange runs Auctions. Implementations must be threadsafe, and will be shared across many goroutines. type Exchange interface { // HoldAuction executes an OpenRTB v2.5 Auction. - HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, usersyncs IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *DebugLog) (*openrtb.BidResponse, error) + HoldAuction(ctx context.Context, r AuctionRequest, debugLog *DebugLog) (*openrtb.BidResponse, error) } // IdFetcher can find the user's ID for a specific Bidder. type IdFetcher interface { // GetId returns the ID for the bidder. The boolean will be true if the ID exists, and false otherwise. GetId(bidder openrtb_ext.BidderName) (string, bool) + LiveSyncCount() int } type exchange struct { @@ -49,8 +59,8 @@ type exchange struct { gDPR gdpr.Permissions currencyConverter *currencies.RateConverter UsersyncIfAmbiguous bool - defaultTTLs config.DefaultTTLs privacyConfig config.Privacy + categoriesFetcher stored_requests.CategoryFetcher } // Container to pass out response ext data from the GetAllBids goroutines back into the main thread @@ -68,7 +78,7 @@ type bidResponseWrapper struct { bidder openrtb_ext.BidderName } -func NewExchange(client *http.Client, cache prebid_cache_client.Client, cfg *config.Configuration, metricsEngine pbsmetrics.MetricsEngine, infos adapters.BidderInfos, gDPR gdpr.Permissions, currencyConverter *currencies.RateConverter) Exchange { +func NewExchange(client *http.Client, cache prebid_cache_client.Client, cfg *config.Configuration, metricsEngine pbsmetrics.MetricsEngine, infos adapters.BidderInfos, gDPR gdpr.Permissions, currencyConverter *currencies.RateConverter, categoriesFetcher stored_requests.CategoryFetcher) Exchange { e := new(exchange) e.adapterMap = newAdapterMap(client, cfg, infos, metricsEngine) @@ -78,101 +88,79 @@ func NewExchange(client *http.Client, cache prebid_cache_client.Client, cfg *con e.gDPR = gDPR e.currencyConverter = currencyConverter e.UsersyncIfAmbiguous = cfg.GDPR.UsersyncIfAmbiguous - e.defaultTTLs = cfg.CacheURL.DefaultTTLs e.privacyConfig = config.Privacy{ CCPA: cfg.CCPA, GDPR: cfg.GDPR, LMT: cfg.LMT, } + e.categoriesFetcher = categoriesFetcher return e } -func (e *exchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidRequest, usersyncs IdFetcher, labels pbsmetrics.Labels, categoriesFetcher *stored_requests.CategoryFetcher, debugLog *DebugLog) (*openrtb.BidResponse, error) { - debug := false - if bidRequest.Ext != nil { - var requestExt openrtb_ext.ExtRequest - err := json.Unmarshal(bidRequest.Ext, &requestExt) - if err != nil { - return nil, fmt.Errorf("Error decoding Request.ext : %s", err.Error()) - } +type AuctionRequest struct { + BidRequest *openrtb.BidRequest + Account config.Account + UserSyncs IdFetcher + RequestType pbsmetrics.RequestType - if requestExt.Prebid.Debug == 1 { - debug = true - } - } + // LegacyLabels is included here for temporary compatability with cleanOpenRTBRequests + // in HoldAuction until we get to factoring it away. Do not use for anything new. + LegacyLabels pbsmetrics.Labels +} - // Snapshot of resolved bid request for debug if test request - resolvedRequest, err := buildResolvedRequest(bidRequest, debug) +func (e *exchange) HoldAuction(ctx context.Context, r AuctionRequest, debugLog *DebugLog) (*openrtb.BidResponse, error) { + var err error + requestExt, err := extractBidRequestExt(r.BidRequest) if err != nil { - glog.Errorf("Error marshalling bid request for debug: %v", err) + return nil, err } - for _, impInRequest := range bidRequest.Imp { - var impLabels pbsmetrics.ImpLabels = pbsmetrics.ImpLabels{ - BannerImps: impInRequest.Banner != nil, - VideoImps: impInRequest.Video != nil, - AudioImps: impInRequest.Audio != nil, - NativeImps: impInRequest.Native != nil, - } - e.me.RecordImps(impLabels) + cacheInstructions := getExtCacheInstructions(requestExt) + targData := getExtTargetData(requestExt, &cacheInstructions) + if targData != nil { + _, targData.cacheHost, targData.cachePath = e.cache.GetExtCacheData() + } + + debugInfo := getDebugInfo(r.BidRequest, requestExt) + if debugInfo { + ctx = e.makeDebugContext(ctx, debugInfo) } + bidAdjustmentFactors := getExtBidAdjustmentFactors(requestExt) + + recordImpMetrics(r.BidRequest, e.me) + + // Make our best guess if GDPR applies + usersyncIfAmbiguous := e.parseUsersyncIfAmbiguous(r.BidRequest) + // Slice of BidRequests, each a copy of the original cleaned to only contain bidder data for the named bidder blabels := make(map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels) - cleanRequests, aliases, errs := cleanOpenRTBRequests(ctx, bidRequest, usersyncs, blabels, labels, e.gDPR, e.UsersyncIfAmbiguous, e.privacyConfig) + cleanRequests, aliases, privacyLabels, errs := cleanOpenRTBRequests(ctx, r.BidRequest, requestExt, r.UserSyncs, blabels, r.LegacyLabels, e.gDPR, usersyncIfAmbiguous, e.privacyConfig, &r.Account) + + e.me.RecordRequestPrivacy(privacyLabels) // List of bidders we have requests for. liveAdapters := listBiddersWithRequests(cleanRequests) - // Process the request to check for targeting parameters. - var targData *targetData - shouldCacheBids := false - shouldCacheVAST := false - var bidAdjustmentFactors map[string]float64 - var requestExt openrtb_ext.ExtRequest - if len(bidRequest.Ext) > 0 { - err := json.Unmarshal(bidRequest.Ext, &requestExt) - if err != nil { - return nil, fmt.Errorf("Error decoding Request.ext : %s", err.Error()) - } - bidAdjustmentFactors = requestExt.Prebid.BidAdjustmentFactors - if requestExt.Prebid.Cache != nil { - shouldCacheBids = requestExt.Prebid.Cache.Bids != nil - shouldCacheVAST = requestExt.Prebid.Cache.VastXML != nil - } - - if requestExt.Prebid.Targeting != nil { - targData = &targetData{ - priceGranularity: requestExt.Prebid.Targeting.PriceGranularity, - includeWinners: requestExt.Prebid.Targeting.IncludeWinners, - includeBidderKeys: requestExt.Prebid.Targeting.IncludeBidderKeys, - includeCacheBids: shouldCacheBids, - includeCacheVast: shouldCacheVAST, - } - targData.cacheHost, targData.cachePath = e.cache.GetExtCacheData() - } - } - // If we need to cache bids, then it will take some time to call prebid cache. // We should reduce the amount of time the bidders have, to compensate. - auctionCtx, cancel := e.makeAuctionContext(ctx, shouldCacheBids) //Why no context for `shouldCacheVast`? + auctionCtx, cancel := e.makeAuctionContext(ctx, cacheInstructions.cacheBids) defer cancel() // Get currency rates conversions for the auction conversions := e.currencyConverter.Rates() - adapterBids, adapterExtra, anyBidsReturned := e.getAllBids(auctionCtx, cleanRequests, aliases, bidAdjustmentFactors, blabels, conversions, debug) + adapterBids, adapterExtra, anyBidsReturned := e.getAllBids(auctionCtx, cleanRequests, aliases, bidAdjustmentFactors, blabels, conversions) - var auc *auction = nil - var bidResponseExt *openrtb_ext.ExtBidResponse = nil + var auc *auction + var cacheErrs []error if anyBidsReturned { var bidCategory map[string]string //If includebrandcategory is present in ext then CE feature is on. if requestExt.Prebid.Targeting != nil && requestExt.Prebid.Targeting.IncludeBrandCategory != nil { - var err error var rejections []string - bidCategory, adapterBids, rejections, err = applyCategoryMapping(ctx, bidRequest, requestExt, adapterBids, *categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err = applyCategoryMapping(ctx, r.BidRequest, requestExt, adapterBids, e.categoriesFetcher, targData) if err != nil { return nil, fmt.Errorf("Error in category mapping : %s", err.Error()) } @@ -181,56 +169,86 @@ func (e *exchange) HoldAuction(ctx context.Context, bidRequest *openrtb.BidReque } } - auc = newAuction(adapterBids, len(bidRequest.Imp)) - if targData != nil { + // A non-nil auction is only needed if targeting is active. (It is used below this block to extract cache keys) + auc = newAuction(adapterBids, len(r.BidRequest.Imp), targData.preferDeals) auc.setRoundedPrices(targData.priceGranularity) if requestExt.Prebid.SupportDeals { - dealErrs := applyDealSupport(bidRequest, auc, bidCategory) + dealErrs := applyDealSupport(r.BidRequest, auc, bidCategory) errs = append(errs, dealErrs...) } - if debugLog != nil && debugLog.Enabled { - bidResponseExt = e.makeExtBidResponse(adapterBids, adapterExtra, bidRequest, resolvedRequest, debug, errs) - if bidRespExtBytes, err := json.Marshal(bidResponseExt); err == nil { - debugLog.Data.Response = string(bidRespExtBytes) - } else { - debugLog.Data.Response = "Unable to marshal response ext for debugging" - errs = append(errs, errors.New(debugLog.Data.Response)) - } - } - - cacheErrs := auc.doCache(ctx, e.cache, targData, bidRequest, 60, &e.defaultTTLs, bidCategory, debugLog) + cacheErrs := auc.doCache(ctx, e.cache, targData, r.BidRequest, 60, &r.Account.CacheTTL, bidCategory, debugLog) if len(cacheErrs) > 0 { errs = append(errs, cacheErrs...) } - targData.setTargeting(auc, bidRequest.App != nil, bidCategory) + targData.setTargeting(auc, r.BidRequest.App != nil, bidCategory) - // Ensure caching errors are added if the bid response ext has already been created - if bidResponseExt != nil && len(cacheErrs) > 0 { - bidderCacheErrs := errsToBidderErrors(cacheErrs) - bidResponseExt.Errors[openrtb_ext.PrebidExtKey] = append(bidResponseExt.Errors[openrtb_ext.PrebidExtKey], bidderCacheErrs...) - } } + } + bidResponseExt := e.makeExtBidResponse(adapterBids, adapterExtra, r.BidRequest, debugInfo, errs) + + // Ensure caching errors are added in case auc.doCache was called and errors were returned + if len(cacheErrs) > 0 { + bidderCacheErrs := errsToBidderErrors(cacheErrs) + bidResponseExt.Errors[openrtb_ext.PrebidExtKey] = append(bidResponseExt.Errors[openrtb_ext.PrebidExtKey], bidderCacheErrs...) + } + + if debugLog != nil && debugLog.Enabled { + if bidRespExtBytes, err := json.Marshal(bidResponseExt); err == nil { + debugLog.Data.Response = string(bidRespExtBytes) + } else { + debugLog.Data.Response = "Unable to marshal response ext for debugging" + errs = append(errs, err) + } + if !anyBidsReturned { + if rawUUID, err := uuid.NewV4(); err == nil { + debugLog.CacheKey = rawUUID.String() + } else { + errs = append(errs, err) + } + } } // Build the response - return e.buildBidResponse(ctx, liveAdapters, adapterBids, bidRequest, resolvedRequest, adapterExtra, auc, bidResponseExt, debug, errs) + return e.buildBidResponse(ctx, liveAdapters, adapterBids, r.BidRequest, adapterExtra, auc, bidResponseExt, cacheInstructions.returnCreative, errs) } -type DealTierInfo struct { - Prefix string `json:"prefix"` - MinDealTier int `json:"minDealTier"` -} +func (e *exchange) parseUsersyncIfAmbiguous(bidRequest *openrtb.BidRequest) bool { + usersyncIfAmbiguous := e.UsersyncIfAmbiguous + var geo *openrtb.Geo = nil + + if bidRequest.User != nil && bidRequest.User.Geo != nil { + geo = bidRequest.User.Geo + } else if bidRequest.Device != nil && bidRequest.Device.Geo != nil { + geo = bidRequest.Device.Geo + } + if geo != nil { + // If we have a country set, and it is on the list, we assume GDPR applies if not set on the request. + // Otherwise we assume it does not apply as long as it appears "valid" (is 3 characters long). + if _, found := e.privacyConfig.GDPR.EEACountriesMap[strings.ToUpper(geo.Country)]; found { + usersyncIfAmbiguous = false + } else if len(geo.Country) == 3 { + // The country field is formatted properly as a three character country code + usersyncIfAmbiguous = true + } + } -type DealTier struct { - Info *DealTierInfo `json:"dealTier,omitempty"` + return usersyncIfAmbiguous } -type BidderDealTier struct { - DealInfo map[string]*DealTier +func recordImpMetrics(bidRequest *openrtb.BidRequest, metricsEngine pbsmetrics.MetricsEngine) { + for _, impInRequest := range bidRequest.Imp { + var impLabels pbsmetrics.ImpLabels = pbsmetrics.ImpLabels{ + BannerImps: impInRequest.Banner != nil, + VideoImps: impInRequest.Video != nil, + AudioImps: impInRequest.Audio != nil, + NativeImps: impInRequest.Native != nil, + } + metricsEngine.RecordImps(impLabels) + } } // applyDealSupport updates targeting keys with deal prefixes if minimum deal tier exceeded @@ -239,15 +257,13 @@ func applyDealSupport(bidRequest *openrtb.BidRequest, auc *auction, bidCategory impDealMap := getDealTiers(bidRequest) for impID, topBidsPerImp := range auc.winningBidsByBidder { - impDeal := impDealMap[impID].DealInfo + impDeal := impDealMap[impID] for bidder, topBidPerBidder := range topBidsPerImp { - bidderString := bidder.String() - if topBidPerBidder.dealPriority > 0 { - if validateAndNormalizeDealTier(impDeal[bidderString]) { - updateHbPbCatDur(topBidPerBidder, impDeal[bidderString].Info, bidCategory) + if validateDealTier(impDeal[bidder]) { + updateHbPbCatDur(topBidPerBidder, impDeal[bidder], bidCategory) } else { - errs = append(errs, fmt.Errorf("dealTier configuration invalid for bidder '%s', imp ID '%s'", bidderString, impID)) + errs = append(errs, fmt.Errorf("dealTier configuration invalid for bidder '%s', imp ID '%s'", string(bidder), impID)) } } } @@ -257,36 +273,29 @@ func applyDealSupport(bidRequest *openrtb.BidRequest, auc *auction, bidCategory } // getDealTiers creates map of impression to bidder deal tier configuration -func getDealTiers(bidRequest *openrtb.BidRequest) map[string]*BidderDealTier { - impDealMap := make(map[string]*BidderDealTier) +func getDealTiers(bidRequest *openrtb.BidRequest) map[string]openrtb_ext.DealTierBidderMap { + impDealMap := make(map[string]openrtb_ext.DealTierBidderMap) for _, imp := range bidRequest.Imp { - var bidderDealTier BidderDealTier - err := json.Unmarshal(imp.Ext, &bidderDealTier.DealInfo) + dealTierBidderMap, err := openrtb_ext.ReadDealTiersFromImp(imp) if err != nil { continue } - - impDealMap[imp.ID] = &bidderDealTier + impDealMap[imp.ID] = dealTierBidderMap } return impDealMap } -func validateAndNormalizeDealTier(impDeal *DealTier) bool { - if impDeal == nil || impDeal.Info == nil { - return false - } - // Remove whitespace from prefix before checking if it can be used - impDeal.Info.Prefix = strings.ReplaceAll(impDeal.Info.Prefix, " ", "") - return len(impDeal.Info.Prefix) > 0 && impDeal.Info.MinDealTier > 0 +func validateDealTier(dealTier openrtb_ext.DealTier) bool { + return len(dealTier.Prefix) > 0 && dealTier.MinDealTier > 0 } -func updateHbPbCatDur(bid *pbsOrtbBid, dealTierInfo *DealTierInfo, bidCategory map[string]string) { - if bid.dealPriority >= dealTierInfo.MinDealTier { +func updateHbPbCatDur(bid *pbsOrtbBid, dealTier openrtb_ext.DealTier, bidCategory map[string]string) { + if bid.dealPriority >= dealTier.MinDealTier { + prefixTier := fmt.Sprintf("%s%d_", dealTier.Prefix, bid.dealPriority) bid.dealTierSatisfied = true - prefixTier := fmt.Sprintf("%s%d_", dealTierInfo.Prefix, bid.dealPriority) if oldCatDur, ok := bidCategory[bid.bid.ID]; ok { oldCatDurSplit := strings.SplitAfterN(oldCatDur, "_", 2) oldCatDurSplit[0] = prefixTier @@ -297,6 +306,11 @@ func updateHbPbCatDur(bid *pbsOrtbBid, dealTierInfo *DealTierInfo, bidCategory m } } +func (e *exchange) makeDebugContext(ctx context.Context, debugInfo bool) (debugCtx context.Context) { + debugCtx = context.WithValue(ctx, DebugContextKey, debugInfo) + return +} + func (e *exchange) makeAuctionContext(ctx context.Context, needsCache bool) (auctionCtx context.Context, cancel context.CancelFunc) { auctionCtx = ctx cancel = func() {} @@ -309,7 +323,7 @@ func (e *exchange) makeAuctionContext(ctx context.Context, needsCache bool) (auc } // This piece sends all the requests to the bidder adapters and gathers the results. -func (e *exchange) getAllBids(ctx context.Context, cleanRequests map[openrtb_ext.BidderName]*openrtb.BidRequest, aliases map[string]string, bidAdjustments map[string]float64, blabels map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels, conversions currencies.Conversions, debug bool) (map[openrtb_ext.BidderName]*pbsOrtbSeatBid, map[openrtb_ext.BidderName]*seatResponseExtra, bool) { +func (e *exchange) getAllBids(ctx context.Context, cleanRequests map[openrtb_ext.BidderName]*openrtb.BidRequest, aliases map[string]string, bidAdjustments map[string]float64, blabels map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels, conversions currencies.Conversions) (map[openrtb_ext.BidderName]*pbsOrtbSeatBid, map[openrtb_ext.BidderName]*seatResponseExtra, bool) { // Set up pointers to the bid results adapterBids := make(map[openrtb_ext.BidderName]*pbsOrtbSeatBid, len(cleanRequests)) adapterExtra := make(map[openrtb_ext.BidderName]*seatResponseExtra, len(cleanRequests)) @@ -340,7 +354,7 @@ func (e *exchange) getAllBids(ctx context.Context, cleanRequests map[openrtb_ext } var reqInfo adapters.ExtraRequestInfo reqInfo.PbsEntryPoint = bidlabels.RType - bids, err := e.adapterMap[coreBidder].requestBid(ctx, request, aName, adjustmentFactor, conversions, &reqInfo, debug) + bids, err := e.adapterMap[coreBidder].requestBid(ctx, request, aName, adjustmentFactor, conversions, &reqInfo) // Add in time reporting elapsed := time.Since(start) @@ -389,6 +403,7 @@ func (e *exchange) getAllBids(ctx context.Context, cleanRequests map[openrtb_ext bidsFound = true bidIDsCollision = recordAdaptorDuplicateBidIDs(e.me, adapterBids) } + } if bidIDsCollision { // record this request count this request if bid collision is detected @@ -466,8 +481,9 @@ func errsToBidderErrors(errs []error) []openrtb_ext.ExtBidderError { } // This piece takes all the bids supplied by the adapters and crafts an openRTB response to send back to the requester -func (e *exchange) buildBidResponse(ctx context.Context, liveAdapters []openrtb_ext.BidderName, adapterBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, bidRequest *openrtb.BidRequest, resolvedRequest json.RawMessage, adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, auc *auction, bidResponseExt *openrtb_ext.ExtBidResponse, debug bool, errList []error) (*openrtb.BidResponse, error) { +func (e *exchange) buildBidResponse(ctx context.Context, liveAdapters []openrtb_ext.BidderName, adapterBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, bidRequest *openrtb.BidRequest, adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, auc *auction, bidResponseExt *openrtb_ext.ExtBidResponse, returnCreative bool, errList []error) (*openrtb.BidResponse, error) { bidResponse := new(openrtb.BidResponse) + var err error bidResponse.ID = bidRequest.ID if len(liveAdapters) == 0 { @@ -481,7 +497,7 @@ func (e *exchange) buildBidResponse(ctx context.Context, liveAdapters []openrtb_ for _, a := range liveAdapters { //while processing every single bib, do we need to handle categories here? if adapterBids[a] != nil && len(adapterBids[a].bids) > 0 { - sb := e.makeSeatBid(adapterBids[a], a, adapterExtra, auc) + sb := e.makeSeatBid(adapterBids[a], a, adapterExtra, auc, returnCreative) seatBids = append(seatBids, *sb) bidResponse.Cur = adapterBids[a].currency } @@ -489,33 +505,42 @@ func (e *exchange) buildBidResponse(ctx context.Context, liveAdapters []openrtb_ bidResponse.SeatBid = seatBids - if bidResponseExt == nil { - bidResponseExt = e.makeExtBidResponse(adapterBids, adapterExtra, bidRequest, resolvedRequest, debug, errList) - } + bidResponse.Ext, err = encodeBidResponseExt(bidResponseExt) + + return bidResponse, err +} + +func encodeBidResponseExt(bidResponseExt *openrtb_ext.ExtBidResponse) ([]byte, error) { buffer := &bytes.Buffer{} enc := json.NewEncoder(buffer) + enc.SetEscapeHTML(false) err := enc.Encode(bidResponseExt) - bidResponse.Ext = buffer.Bytes() - return bidResponse, err + return buffer.Bytes(), err } -func applyCategoryMapping(ctx context.Context, bidRequest *openrtb.BidRequest, requestExt openrtb_ext.ExtRequest, seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, categoriesFetcher stored_requests.CategoryFetcher, targData *targetData) (map[string]string, map[openrtb_ext.BidderName]*pbsOrtbSeatBid, []string, error) { +func applyCategoryMapping(ctx context.Context, bidRequest *openrtb.BidRequest, requestExt *openrtb_ext.ExtRequest, seatBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, categoriesFetcher stored_requests.CategoryFetcher, targData *targetData) (map[string]string, map[openrtb_ext.BidderName]*pbsOrtbSeatBid, []string, error) { res := make(map[string]string) type bidDedupe struct { bidderName openrtb_ext.BidderName bidIndex int bidID string + bidPrice string } dedupe := make(map[string]bidDedupe) + impMap := make(map[string]*openrtb.Imp) + + // applyCategoryMapping doesn't get called unless + // requestExt.Prebid.Targeting != nil && requestExt.Prebid.Targeting.IncludeBrandCategory != nil brandCatExt := requestExt.Prebid.Targeting.IncludeBrandCategory //If ext.prebid.targeting.includebrandcategory is present in ext then competitive exclusion feature is on. var includeBrandCategory = brandCatExt != nil //if not present - category will no be appended + appendBidderNames := requestExt.Prebid.Targeting.AppendBidderNames var primaryAdServer string var publisher string @@ -586,7 +611,7 @@ func applyCategoryMapping(ctx context.Context, bidRequest *openrtb.BidRequest, r // TODO: consider should we remove bids with zero duration here? - pb, _ = GetCpmStringValue(bid.bid.Price, targData.priceGranularity) + pb = GetPriceBucket(bid.bid.Price, targData.priceGranularity) newDur := duration if len(requestExt.Prebid.Targeting.DurationRangeSec) > 0 { @@ -613,16 +638,39 @@ func applyCategoryMapping(ctx context.Context, bidRequest *openrtb.BidRequest, r } var categoryDuration string + var dupeKey string if brandCatExt.WithCategory { categoryDuration = fmt.Sprintf("%s_%s_%ds", pb, category, newDur) + dupeKey = category } else { categoryDuration = fmt.Sprintf("%s_%ds", pb, newDur) + dupeKey = categoryDuration + } + + if appendBidderNames { + categoryDuration = fmt.Sprintf("%s_%s", categoryDuration, bidderName.String()) } if false == brandCatExt.SkipDedup { - if dupe, ok := dedupe[categoryDuration]; ok { - // 50% chance for either bid with duplicate categoryDuration values to be kept - if rand.Intn(100) < 50 { + if dupe, ok := dedupe[dupeKey]; ok { + + dupeBidPrice, err := strconv.ParseFloat(dupe.bidPrice, 64) + if err != nil { + dupeBidPrice = 0 + } + currBidPrice, err := strconv.ParseFloat(pb, 64) + if err != nil { + currBidPrice = 0 + } + if dupeBidPrice == currBidPrice { + if rand.Intn(100) < 50 { + dupeBidPrice = -1 + } else { + currBidPrice = -1 + } + } + + if dupeBidPrice < currBidPrice { if dupe.bidderName == bidderName { // An older bid from the current bidder bidsToRemove = append(bidsToRemove, dupe.bidIndex) @@ -631,7 +679,7 @@ func applyCategoryMapping(ctx context.Context, bidRequest *openrtb.BidRequest, r // An older bid from a different seatBid we've already finished with oldSeatBid := (seatBids)[dupe.bidderName] if len(oldSeatBid.bids) == 1 { - seatBidsToRemove = append(seatBidsToRemove, bidderName) + seatBidsToRemove = append(seatBidsToRemove, dupe.bidderName) rejections = updateRejections(rejections, dupe.bidID, "Bid was deduplicated") } else { oldSeatBid.bids = append(oldSeatBid.bids[:dupe.bidIndex], oldSeatBid.bids[dupe.bidIndex+1:]...) @@ -645,9 +693,8 @@ func applyCategoryMapping(ctx context.Context, bidRequest *openrtb.BidRequest, r continue } } - dedupe[categoryDuration] = bidDedupe{bidderName: bidderName, bidIndex: bidInd, bidID: bidID} + dedupe[dupeKey] = bidDedupe{bidderName: bidderName, bidIndex: bidInd, bidID: bidID, bidPrice: pb} } - res[bidID] = categoryDuration } @@ -691,24 +738,22 @@ func getPrimaryAdServer(adServerId int) (string, error) { } // Extract all the data from the SeatBids and build the ExtBidResponse -func (e *exchange) makeExtBidResponse(adapterBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, req *openrtb.BidRequest, resolvedRequest json.RawMessage, debug bool, errList []error) *openrtb_ext.ExtBidResponse { +func (e *exchange) makeExtBidResponse(adapterBids map[openrtb_ext.BidderName]*pbsOrtbSeatBid, adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, req *openrtb.BidRequest, debugInfo bool, errList []error) *openrtb_ext.ExtBidResponse { bidResponseExt := &openrtb_ext.ExtBidResponse{ Errors: make(map[openrtb_ext.BidderName][]openrtb_ext.ExtBidderError, len(adapterBids)), ResponseTimeMillis: make(map[openrtb_ext.BidderName]int, len(adapterBids)), RequestTimeoutMillis: req.TMax, } - if debug { + if debugInfo { bidResponseExt.Debug = &openrtb_ext.ExtResponseDebug{ - HttpCalls: make(map[openrtb_ext.BidderName][]*openrtb_ext.ExtHttpCall), - } - if err := json.Unmarshal(resolvedRequest, &bidResponseExt.Debug.ResolvedRequest); err != nil { - glog.Errorf("Error unmarshalling bid request snapshot: %v", err) + HttpCalls: make(map[openrtb_ext.BidderName][]*openrtb_ext.ExtHttpCall), + ResolvedRequest: req, } } for bidderName, responseExtra := range adapterExtra { - if debug { + if debugInfo { bidResponseExt.Debug.HttpCalls[bidderName] = responseExtra.HttpCalls } // Only make an entry for bidder errors if the bidder reported any. @@ -727,7 +772,7 @@ func (e *exchange) makeExtBidResponse(adapterBids map[openrtb_ext.BidderName]*pb // Return an openrtb seatBid for a bidder // BuildBidResponse is responsible for ensuring nil bid seatbids are not included -func (e *exchange) makeSeatBid(adapterBid *pbsOrtbSeatBid, adapter openrtb_ext.BidderName, adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, auc *auction) *openrtb.SeatBid { +func (e *exchange) makeSeatBid(adapterBid *pbsOrtbSeatBid, adapter openrtb_ext.BidderName, adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, auc *auction, returnCreative bool) *openrtb.SeatBid { seatBid := new(openrtb.SeatBid) seatBid.Seat = adapter.String() // Prebid cannot support roadblocking @@ -750,7 +795,7 @@ func (e *exchange) makeSeatBid(adapterBid *pbsOrtbSeatBid, adapter openrtb_ext.B } var errList []error - seatBid.Bid, errList = e.makeBid(adapterBid.bids, adapter, auc) + seatBid.Bid, errList = e.makeBid(adapterBid.bids, auc, returnCreative) if len(errList) > 0 { adapterExtra[adapter].Errors = append(adapterExtra[adapter].Errors, errsToBidderErrors(errList)...) } @@ -759,7 +804,7 @@ func (e *exchange) makeSeatBid(adapterBid *pbsOrtbSeatBid, adapter openrtb_ext.B } // Create the Bid array inside of SeatBid -func (e *exchange) makeBid(Bids []*pbsOrtbBid, adapter openrtb_ext.BidderName, auc *auction) ([]openrtb.Bid, []error) { +func (e *exchange) makeBid(Bids []*pbsOrtbBid, auc *auction, returnCreative bool) ([]openrtb.Bid, []error) { bids := make([]openrtb.Bid, 0, len(Bids)) errList := make([]error, 0, 1) for _, thisBid := range Bids { @@ -784,6 +829,9 @@ func (e *exchange) makeBid(Bids []*pbsOrtbBid, adapter openrtb_ext.BidderName, a } else { bids = append(bids, *thisBid.bid) bids[len(bids)-1].Ext = ext + if !returnCreative { + bids[len(bids)-1].AdM = "" + } } } return bids, errList @@ -791,32 +839,50 @@ func (e *exchange) makeBid(Bids []*pbsOrtbBid, adapter openrtb_ext.BidderName, a // If bid got cached inside `(a *auction) doCache(ctx context.Context, cache prebid_cache_client.Client, targData *targetData, bidRequest *openrtb.BidRequest, ttlBuffer int64, defaultTTLs *config.DefaultTTLs, bidCategory map[string]string)`, // a UUID should be found inside `a.cacheIds` or `a.vastCacheIds`. This function returns the UUID along with the internal cache URL -func (e *exchange) getBidCacheInfo(bid *pbsOrtbBid, auc *auction) (openrtb_ext.ExtBidPrebidCacheBids, bool) { - var cacheInfo openrtb_ext.ExtBidPrebidCacheBids - var cacheUUID string - var found bool = false - - if auc != nil { - var extCacheHost, extCachePath string - if cacheUUID, found = auc.cacheIds[bid.bid]; found { - cacheInfo.CacheId = cacheUUID - extCacheHost, extCachePath = e.cache.GetExtCacheData() - cacheInfo.Url = extCacheHost + extCachePath + "?uuid=" + cacheUUID - } else if cacheUUID, found = auc.vastCacheIds[bid.bid]; found { - cacheInfo.CacheId = cacheUUID - extCacheHost, extCachePath = e.cache.GetExtCacheData() - cacheInfo.Url = extCacheHost + extCachePath + "?uuid=" + cacheUUID +func (e *exchange) getBidCacheInfo(bid *pbsOrtbBid, auction *auction) (cacheInfo openrtb_ext.ExtBidPrebidCacheBids, found bool) { + uuid, found := findCacheID(bid, auction) + + if found { + cacheInfo.CacheId = uuid + cacheInfo.Url = buildCacheURL(e.cache, uuid) + } + + return +} + +func findCacheID(bid *pbsOrtbBid, auction *auction) (string, bool) { + if bid != nil && bid.bid != nil && auction != nil { + if id, found := auction.cacheIds[bid.bid]; found { + return id, true + } + + if id, found := auction.vastCacheIds[bid.bid]; found { + return id, true } } - return cacheInfo, found + + return "", false } -// Returns a snapshot of resolved bid request for debug if test field is set in the incomming request -func buildResolvedRequest(bidRequest *openrtb.BidRequest, debug bool) (json.RawMessage, error) { - if debug { - return json.Marshal(bidRequest) +func buildCacheURL(cache prebid_cache_client.Client, uuid string) string { + scheme, host, path := cache.GetExtCacheData() + + if host == "" || path == "" { + return "" } - return nil, nil + + query := url.Values{"uuid": []string{uuid}} + cacheURL := url.URL{ + Scheme: scheme, + Host: host, + Path: path, + RawQuery: query.Encode(), + } + cacheURL.Query() + + // URLs without a scheme will begin with //, in which case we + // want to trim it off to keep compatbile with current behavior. + return strings.TrimPrefix(cacheURL.String(), "//") } func listBiddersWithRequests(cleanRequests map[openrtb_ext.BidderName]*openrtb.BidRequest) []openrtb_ext.BidderName { diff --git a/exchange/exchange_test.go b/exchange/exchange_test.go index aa48b9b71cc..6f9e9cdfa0b 100644 --- a/exchange/exchange_test.go +++ b/exchange/exchange_test.go @@ -16,18 +16,18 @@ import ( "time" "github.com/PubMatic-OpenWrap/prebid-server/adapters" - "github.com/PubMatic-OpenWrap/prebid-server/currencies" - "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" - "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" - "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/file_fetcher" - - "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/currencies" "github.com/PubMatic-OpenWrap/prebid-server/gdpr" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" metricsConf "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics/config" + metricsConfig "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics/config" pbc "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/file_fetcher" + + "github.com/PubMatic-OpenWrap/openrtb" "github.com/buger/jsonparser" "github.com/rcrowley/go-metrics" "github.com/stretchr/testify/assert" @@ -48,9 +48,13 @@ func TestNewExchange(t *testing.T) { ExpectedTimeMillis: 20, }, Adapters: blankAdapterConfig(openrtb_ext.BidderList()), + GDPR: config.GDPR{ + EEACountries: []string{"FIN", "FRA", "GUF"}, + }, } - e := NewExchange(server.Client(), nil, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), knownAdapters, config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()).(*exchange) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e := NewExchange(server.Client(), nil, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), knownAdapters, config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, nilCategoryFetcher{}).(*exchange) for _, bidderName := range knownAdapters { if _, ok := e.adapterMap[bidderName]; !ok { t.Errorf("NewExchange produced an Exchange without bidder %s", bidderName) @@ -87,7 +91,8 @@ func TestCharacterEscape(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(handlerNoBidServer)) defer server.Close() - e := NewExchange(server.Client(), nil, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()).(*exchange) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e := NewExchange(server.Client(), nil, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, nilCategoryFetcher{}).(*exchange) /* 3) Build all the parameters e.buildBidResponse(ctx.Background(), liveA... ) needs */ //liveAdapters []openrtb_ext.BidderName, @@ -113,9 +118,6 @@ func TestCharacterEscape(t *testing.T) { Ext: json.RawMessage(`{"id": "some-request-id","site": {"page": "prebid.org"},"imp": [{"id": "some-impression-id","banner": {"format": [{"w": 300,"h": 250},{"w": 300,"h": 600}]},"ext": {"appnexus": {"placementId": 1}}}],"tmax": 500}`), } - //resolvedRequest json.RawMessage - resolvedRequest := json.RawMessage(`{"id": "some-request-id","site": {"page": "prebid.org"},"imp": [{"id": "some-impression-id","banner": {"format": [{"w": 300,"h": 250},{"w": 300,"h": 600}]},"ext": {"appnexus": {"placementId": 1}}}],"tmax": 500}`) - //adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, adapterExtra := make(map[openrtb_ext.BidderName]*seatResponseExtra, 1) adapterExtra["appnexus"] = &seatResponseExtra{ @@ -127,7 +129,7 @@ func TestCharacterEscape(t *testing.T) { var errList []error /* 4) Build bid response */ - bidResp, err := e.buildBidResponse(context.Background(), liveAdapters, adapterBids, bidRequest, resolvedRequest, adapterExtra, nil, nil, false, errList) + bidResp, err := e.buildBidResponse(context.Background(), liveAdapters, adapterBids, bidRequest, adapterExtra, nil, nil, true, errList) /* 5) Assert we have no errors and one '&' character as we are supposed to */ if err != nil { @@ -141,9 +143,342 @@ func TestCharacterEscape(t *testing.T) { } } -func TestGetBidCacheInfo(t *testing.T) { +// TestDebugBehaviour asserts the HttpCalls object is included inside the json "debug" field of the bidResponse extension when the +// openrtb.BidRequest "Test" value is set to 1 or the openrtb.BidRequest.Ext.Debug boolean field is set to true +func TestDebugBehaviour(t *testing.T) { + + // Define test cases + type inTest struct { + test int8 + debug bool + } + type outTest struct { + debugInfoIncluded bool + } + type aTest struct { + desc string + in inTest + out outTest + } + testCases := []aTest{ + { + desc: "test flag equals zero, ext debug flag false, no debug info expected", + in: inTest{test: 0, debug: false}, + out: outTest{debugInfoIncluded: false}, + }, + { + desc: "test flag equals zero, ext debug flag true, debug info expected", + in: inTest{test: 0, debug: true}, + out: outTest{debugInfoIncluded: true}, + }, + { + desc: "test flag equals 1, ext debug flag false, debug info expected", + in: inTest{test: 1, debug: false}, + out: outTest{debugInfoIncluded: true}, + }, + { + desc: "test flag equals 1, ext debug flag true, debug info expected", + in: inTest{test: 1, debug: true}, + out: outTest{debugInfoIncluded: true}, + }, + { + desc: "test flag not equal to 0 nor 1, ext debug flag false, no debug info expected", + in: inTest{test: 2, debug: false}, + out: outTest{debugInfoIncluded: false}, + }, + { + desc: "test flag not equal to 0 nor 1, ext debug flag true, debug info expected", + in: inTest{test: -1, debug: true}, + out: outTest{debugInfoIncluded: true}, + }, + } + + // Set up test + noBidServer := func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(204) + } + server := httptest.NewServer(http.HandlerFunc(noBidServer)) + defer server.Close() + + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + + bidRequest := &openrtb.BidRequest{ + ID: "some-request-id", + Imp: []openrtb.Imp{{ + ID: "some-impression-id", + Banner: &openrtb.Banner{Format: []openrtb.Format{{W: 300, H: 250}, {W: 300, H: 600}}}, + Ext: json.RawMessage(`{"appnexus": {"placementId": 1}}`), + }}, + Site: &openrtb.Site{Page: "prebid.org", Ext: json.RawMessage(`{"amp":0}`)}, + Device: &openrtb.Device{UA: "curl/7.54.0", IP: "::1"}, + AT: 1, + TMax: 500, + } + + bidderImpl := &goodSingleBidder{ + httpRequest: &adapters.RequestData{ + Method: "POST", + Uri: server.URL, + Body: []byte("{\"key\":\"val\"}"), + Headers: http.Header{}, + }, + bidResponse: &adapters.BidderResponse{}, + } + + e := new(exchange) + e.adapterMap = map[openrtb_ext.BidderName]adaptedBidder{ + openrtb_ext.BidderAppnexus: adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus), + } + e.cache = &wellBehavedCache{} + e.me = &metricsConf.DummyMetricsEngine{} + e.gDPR = gdpr.AlwaysAllow{} + e.currencyConverter = currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e.categoriesFetcher = categoriesFetcher + + // Run tests + for _, test := range testCases { + bidRequest.Test = test.in.test + + if test.in.debug { + bidRequest.Ext = json.RawMessage(`{"prebid":{"debug":true}}`) + } else { + bidRequest.Ext = nil + } + + auctionRequest := AuctionRequest{ + BidRequest: bidRequest, + Account: config.Account{}, + UserSyncs: &emptyUsersync{}, + } + + // Run test + outBidResponse, err := e.HoldAuction(context.Background(), auctionRequest, nil) + + // Assert no HoldAuction error + assert.NoErrorf(t, err, "%s. ex.HoldAuction returned an error: %v \n", test.desc, err) + assert.NotNilf(t, outBidResponse.Ext, "%s. outBidResponse.Ext should not be nil \n", test.desc) + + actualExt := &openrtb_ext.ExtBidResponse{} + err = json.Unmarshal(outBidResponse.Ext, actualExt) + assert.NoErrorf(t, err, "%s. \"ext\" JSON field could not be unmarshaled. err: \"%v\" \n outBidResponse.Ext: \"%s\" \n", test.desc, err, outBidResponse.Ext) + + if test.out.debugInfoIncluded { + assert.NotNilf(t, actualExt, "%s. ext.debug field is expected to be included in this outBidResponse.Ext and not be nil. outBidResponse.Ext.Debug = %v \n", test.desc, actualExt.Debug) + + // Assert "Debug fields + assert.Greater(t, len(actualExt.Debug.HttpCalls), 0, "%s. ext.debug.httpcalls array should not be empty\n", test.desc) + assert.Equal(t, server.URL, actualExt.Debug.HttpCalls["appnexus"][0].Uri, "%s. ext.debug.httpcalls array should not be empty\n", test.desc) + assert.NotNilf(t, actualExt.Debug.ResolvedRequest, "%s. ext.debug.resolvedrequest field is expected to be included in this outBidResponse.Ext and not be nil. outBidResponse.Ext.Debug = %v \n", test.desc, actualExt.Debug) + + // If not nil, assert bid extension + if test.in.debug { + diffJson(t, test.desc, bidRequest.Ext, actualExt.Debug.ResolvedRequest.Ext) + } + } + } +} + +func TestReturnCreativeEndToEnd(t *testing.T) { + sampleAd := "" + + // Define test cases + type aTest struct { + desc string + inExt json.RawMessage + outAdM string + } + testGroups := []struct { + groupDesc string + testCases []aTest + expectError bool + }{ + { + groupDesc: "Invalid or malformed bidRequest Ext, expect error in these scenarios", + testCases: []aTest{ + { + desc: "Malformed ext in bidRequest", + inExt: json.RawMessage(`malformed`), + }, + { + desc: "empty cache field", + inExt: json.RawMessage(`{"prebid":{"cache":{}}}`), + }, + }, + expectError: true, + }, + { + groupDesc: "Valid bidRequest Ext but no returnCreative value specified, default to returning creative", + testCases: []aTest{ + { + "Nil ext in bidRequest", + nil, + sampleAd, + }, + { + "empty ext", + json.RawMessage(``), + sampleAd, + }, + { + "bids doesn't come with returnCreative value", + json.RawMessage(`{"prebid":{"cache":{"bids":{}}}}`), + sampleAd, + }, + { + "vast doesn't come with returnCreative value", + json.RawMessage(`{"prebid":{"cache":{"vastXml":{}}}}`), + sampleAd, + }, + }, + }, + { + groupDesc: "Bids field comes with returnCreative value", + testCases: []aTest{ + { + "Bids returnCreative set to true, return ad markup in response", + json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":true}}}}`), + sampleAd, + }, + { + "Bids returnCreative set to false, don't return ad markup in response", + json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":false}}}}`), + "", + }, + }, + }, + { + groupDesc: "Vast field comes with returnCreative value", + testCases: []aTest{ + { + "Vast returnCreative set to true, return ad markup in response", + json.RawMessage(`{"prebid":{"cache":{"vastXml":{"returnCreative":true}}}}`), + sampleAd, + }, + { + "Vast returnCreative set to false, don't return ad markup in response", + json.RawMessage(`{"prebid":{"cache":{"vastXml":{"returnCreative":false}}}}`), + "", + }, + }, + }, + { + groupDesc: "Both Bids and Vast come with their own returnCreative value", + testCases: []aTest{ + { + "Both false, expect empty AdM", + json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":false},"vastXml":{"returnCreative":false}}}}`), + "", + }, + { + "Bids returnCreative is true, expect valid AdM", + json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":true},"vastXml":{"returnCreative":false}}}}`), + sampleAd, + }, + { + "Vast returnCreative is true, expect valid AdM", + json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":false},"vastXml":{"returnCreative":true}}}}`), + sampleAd, + }, + { + "Both field's returnCreative set to true, expect valid AdM", + json.RawMessage(`{"prebid":{"cache":{"bids":{"returnCreative":true},"vastXml":{"returnCreative":true}}}}`), + sampleAd, + }, + }, + }, + } + + // Init an exchange to run an auction from + noBidServer := func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(204) } + server := httptest.NewServer(http.HandlerFunc(noBidServer)) + defer server.Close() + + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + + bidderImpl := &goodSingleBidder{ + httpRequest: &adapters.RequestData{ + Method: "POST", + Uri: server.URL, + Body: []byte("{\"key\":\"val\"}"), + Headers: http.Header{}, + }, + bidResponse: &adapters.BidderResponse{ + Bids: []*adapters.TypedBid{ + { + Bid: &openrtb.Bid{AdM: sampleAd}, + }, + }, + }, + } + + e := new(exchange) + e.adapterMap = map[openrtb_ext.BidderName]adaptedBidder{ + openrtb_ext.BidderAppnexus: adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus), + } + e.cache = &wellBehavedCache{} + e.me = &metricsConf.DummyMetricsEngine{} + e.gDPR = gdpr.AlwaysAllow{} + e.currencyConverter = currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e.categoriesFetcher = categoriesFetcher + + // Define mock incoming bid requeset + mockBidRequest := &openrtb.BidRequest{ + ID: "some-request-id", + Imp: []openrtb.Imp{{ + ID: "some-impression-id", + Banner: &openrtb.Banner{Format: []openrtb.Format{{W: 300, H: 250}, {W: 300, H: 600}}}, + Ext: json.RawMessage(`{"appnexus": {"placementId": 1}}`), + }}, + Site: &openrtb.Site{Page: "prebid.org", Ext: json.RawMessage(`{"amp":0}`)}, + } + + // Run tests + for _, testGroup := range testGroups { + for _, test := range testGroup.testCases { + mockBidRequest.Ext = test.inExt + + auctionRequest := AuctionRequest{ + BidRequest: mockBidRequest, + Account: config.Account{}, + UserSyncs: &emptyUsersync{}, + } + + // Run test + outBidResponse, err := e.HoldAuction(context.Background(), auctionRequest, nil) + + // Assert return error, if any + if testGroup.expectError { + assert.Errorf(t, err, "HoldAuction expected to throw error for: %s - %s. \n", testGroup.groupDesc, test.desc) + continue + } else { + assert.NoErrorf(t, err, "%s: %s. HoldAuction error: %v \n", testGroup.groupDesc, test.desc, err) + } + + // Assert returned bid + if !assert.NotNil(t, outBidResponse, "%s: %s. outBidResponse is nil \n", testGroup.groupDesc, test.desc) { + return + } + if !assert.NotEmpty(t, outBidResponse.SeatBid, "%s: %s. outBidResponse.SeatBid is empty \n", testGroup.groupDesc, test.desc) { + return + } + if !assert.NotEmpty(t, outBidResponse.SeatBid[0].Bid, "%s: %s. outBidResponse.SeatBid[0].Bid is empty \n", testGroup.groupDesc, test.desc) { + return + } + assert.Equal(t, test.outAdM, outBidResponse.SeatBid[0].Bid[0].AdM, "Ad markup string doesn't match in: %s - %s \n", testGroup.groupDesc, test.desc) + } + } +} + +func TestGetBidCacheInfoEndToEnd(t *testing.T) { testUUID := "CACHE_UUID_1234" - testExternalCacheHost := "https://www.externalprebidcache.net" + testExternalCacheScheme := "https" + testExternalCacheHost := "www.externalprebidcache.net" testExternalCachePath := "endpoints/cache" /* 1) An adapter */ @@ -159,8 +494,9 @@ func TestGetBidCacheInfo(t *testing.T) { Host: "www.internalprebidcache.net", }, ExtCacheURL: config.ExternalCache{ - Host: testExternalCacheHost, - Path: testExternalCachePath, + Scheme: testExternalCacheScheme, + Host: testExternalCacheHost, + Path: testExternalCachePath, }, } adapterList := make([]openrtb_ext.BidderName, 0, 2) @@ -171,7 +507,8 @@ func TestGetBidCacheInfo(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(handlerNoBidServer)) defer server.Close() - e := NewExchange(server.Client(), pbc.NewClient(&http.Client{}, &cfg.CacheURL, &cfg.ExtCacheURL, testEngine), cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()).(*exchange) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e := NewExchange(server.Client(), pbc.NewClient(&http.Client{}, &cfg.CacheURL, &cfg.ExtCacheURL, testEngine), cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, nilCategoryFetcher{}).(*exchange) /* 3) Build all the parameters e.buildBidResponse(ctx.Background(), liveA... ) needs */ liveAdapters := []openrtb_ext.BidderName{bidderName} @@ -231,9 +568,6 @@ func TestGetBidCacheInfo(t *testing.T) { }, } - //resolvedRequest json.RawMessage - resolvedRequest := json.RawMessage(`{"id": "some-request-id","site": {"page": "prebid.org"},"imp": [{"id": "some-impression-id","banner": {"format": [{"w": 300,"h": 250},{"w": 300,"h": 600}]},"ext": {"appnexus": {"placementId": 1}}}],"tmax": 500}`) - //adapterExtra map[openrtb_ext.BidderName]*seatResponseExtra, adapterExtra := map[openrtb_ext.BidderName]*seatResponseExtra{ bidderName: { @@ -279,7 +613,7 @@ func TestGetBidCacheInfo(t *testing.T) { var errList []error /* 4) Build bid response */ - bid_resp, err := e.buildBidResponse(context.Background(), liveAdapters, adapterBids, bidRequest, resolvedRequest, adapterExtra, auc, nil, false, errList) + bid_resp, err := e.buildBidResponse(context.Background(), liveAdapters, adapterBids, bidRequest, adapterExtra, auc, nil, true, errList) /* 5) Assert we have no errors and the bid response we expected*/ assert.NoError(t, err, "[TestGetBidCacheInfo] buildBidResponse() threw an error") @@ -290,7 +624,7 @@ func TestGetBidCacheInfo(t *testing.T) { Seat: string(bidderName), Bid: []openrtb.Bid{ { - Ext: json.RawMessage(`{ "prebid": { "cache": { "bids": { "cacheId": "` + testUUID + `", "url": "` + testExternalCacheHost + `/` + testExternalCachePath + `?uuid=` + testUUID + `" }, "key": "", "url": "" }`), + Ext: json.RawMessage(`{ "prebid": { "cache": { "bids": { "cacheId": "` + testUUID + `", "url": "` + testExternalCacheScheme + `://` + testExternalCacheHost + `/` + testExternalCachePath + `?uuid=` + testUUID + `" }, "key": "", "url": "" }`), }, }, }, @@ -305,7 +639,7 @@ func TestGetBidCacheInfo(t *testing.T) { assert.Equal(t, expCacheUUID, cacheUUID, "[TestGetBidCacheInfo] cacheId field in ext should equal \"%s\" \n", expCacheUUID) - // compare cache UUID + // compare cache URL expCacheURL, err := jsonparser.GetString(expectedBidResponse.SeatBid[0].Bid[0].Ext, "prebid", "cache", "bids", "url") assert.NoErrorf(t, err, "[TestGetBidCacheInfo] Error found while trying to json parse the url field from expected build response. Message: %v \n", err) @@ -315,6 +649,199 @@ func TestGetBidCacheInfo(t *testing.T) { assert.Equal(t, expCacheURL, cacheURL, "[TestGetBidCacheInfo] cacheId field in ext should equal \"%s\" \n", expCacheURL) } +func TestBidReturnsCreative(t *testing.T) { + sampleAd := "" + sampleOpenrtbBid := &openrtb.Bid{ID: "some-bid-id", AdM: sampleAd} + + // Define test cases + testCases := []struct { + description string + inReturnCreative bool + expectedCreativeMarkup string + }{ + { + "returnCreative set to true, expect a full creative markup string in returned bid", + true, + sampleAd, + }, + { + "returnCreative set to false, expect empty creative markup string in returned bid", + false, + "", + }, + } + + // Test set up + sampleBids := []*pbsOrtbBid{ + { + bid: sampleOpenrtbBid, + bidType: openrtb_ext.BidTypeBanner, + bidTargets: map[string]string{}, + }, + } + sampleAuction := &auction{cacheIds: map[*openrtb.Bid]string{sampleOpenrtbBid: "CACHE_UUID_1234"}} + + noBidHandler := func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(204) } + server := httptest.NewServer(http.HandlerFunc(noBidHandler)) + defer server.Close() + + bidderImpl := &goodSingleBidder{ + httpRequest: &adapters.RequestData{ + Method: "POST", + Uri: server.URL, + Body: []byte("{\"key\":\"val\"}"), + Headers: http.Header{}, + }, + bidResponse: &adapters.BidderResponse{}, + } + e := new(exchange) + e.adapterMap = map[openrtb_ext.BidderName]adaptedBidder{ + openrtb_ext.BidderAppnexus: adaptBidder(bidderImpl, server.Client(), &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus), + } + e.cache = &wellBehavedCache{} + e.me = &metricsConf.DummyMetricsEngine{} + e.gDPR = gdpr.AlwaysAllow{} + e.currencyConverter = currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + + //Run tests + for _, test := range testCases { + resultingBids, resultingErrs := e.makeBid(sampleBids, sampleAuction, test.inReturnCreative) + + assert.Equal(t, 0, len(resultingErrs), "%s. Test should not return errors \n", test.description) + assert.Equal(t, test.expectedCreativeMarkup, resultingBids[0].AdM, "%s. Ad markup string doesn't match expected \n", test.description) + + var bidExt openrtb_ext.ExtBid + json.Unmarshal(resultingBids[0].Ext, &bidExt) + assert.Equal(t, 0, bidExt.Prebid.DealPriority, "%s. Test should have DealPriority set to 0", test.description) + assert.Equal(t, false, bidExt.Prebid.DealTierSatisfied, "%s. Test should have DealTierSatisfied set to false", test.description) + } +} + +func TestGetBidCacheInfo(t *testing.T) { + bid := &openrtb.Bid{ID: "42"} + testCases := []struct { + description string + scheme string + host string + path string + bid *pbsOrtbBid + auction *auction + expectedFound bool + expectedCacheID string + expectedCacheURL string + }{ + { + description: "JSON Cache ID", + scheme: "https", + host: "prebid.org", + path: "cache", + bid: &pbsOrtbBid{bid: bid}, + auction: &auction{cacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: true, + expectedCacheID: "anyID", + expectedCacheURL: "https://prebid.org/cache?uuid=anyID", + }, + { + description: "VAST Cache ID", + scheme: "https", + host: "prebid.org", + path: "cache", + bid: &pbsOrtbBid{bid: bid}, + auction: &auction{vastCacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: true, + expectedCacheID: "anyID", + expectedCacheURL: "https://prebid.org/cache?uuid=anyID", + }, + { + description: "Cache ID Not Found", + scheme: "https", + host: "prebid.org", + path: "cache", + bid: &pbsOrtbBid{bid: bid}, + auction: &auction{}, + expectedFound: false, + expectedCacheID: "", + expectedCacheURL: "", + }, + { + description: "Scheme Not Provided", + host: "prebid.org", + path: "cache", + bid: &pbsOrtbBid{bid: bid}, + auction: &auction{cacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: true, + expectedCacheID: "anyID", + expectedCacheURL: "prebid.org/cache?uuid=anyID", + }, + { + description: "Host And Path Not Provided - Without Scheme", + bid: &pbsOrtbBid{bid: bid}, + auction: &auction{cacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: true, + expectedCacheID: "anyID", + expectedCacheURL: "", + }, + { + description: "Host And Path Not Provided - With Scheme", + scheme: "https", + bid: &pbsOrtbBid{bid: bid}, + auction: &auction{cacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: true, + expectedCacheID: "anyID", + expectedCacheURL: "", + }, + { + description: "Nil Bid", + scheme: "https", + host: "prebid.org", + path: "cache", + bid: nil, + auction: &auction{cacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: false, + expectedCacheID: "", + expectedCacheURL: "", + }, + { + description: "Nil Embedded Bid", + scheme: "https", + host: "prebid.org", + path: "cache", + bid: &pbsOrtbBid{bid: nil}, + auction: &auction{cacheIds: map[*openrtb.Bid]string{bid: "anyID"}}, + expectedFound: false, + expectedCacheID: "", + expectedCacheURL: "", + }, + { + description: "Nil Auction", + scheme: "https", + host: "prebid.org", + path: "cache", + bid: &pbsOrtbBid{bid: bid}, + auction: nil, + expectedFound: false, + expectedCacheID: "", + expectedCacheURL: "", + }, + } + + for _, test := range testCases { + exchange := &exchange{ + cache: &mockCache{ + scheme: test.scheme, + host: test.host, + path: test.path, + }, + } + + cacheInfo, found := exchange.getBidCacheInfo(test.bid, test.auction) + + assert.Equal(t, test.expectedFound, found, test.description+":found") + assert.Equal(t, test.expectedCacheID, cacheInfo.CacheId, test.description+":id") + assert.Equal(t, test.expectedCacheURL, cacheInfo.Url, test.description+":url") + } +} + func TestBidResponseCurrency(t *testing.T) { // Init objects cfg := &config.Configuration{Adapters: make(map[string]config.Adapter, 1)} @@ -324,7 +851,8 @@ func TestBidResponseCurrency(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(handlerNoBidServer)) defer server.Close() - e := NewExchange(server.Client(), nil, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()).(*exchange) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e := NewExchange(server.Client(), nil, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, nilCategoryFetcher{}).(*exchange) liveAdapters := make([]openrtb_ext.BidderName, 1) liveAdapters[0] = "appnexus" @@ -343,8 +871,6 @@ func TestBidResponseCurrency(t *testing.T) { Ext: json.RawMessage(`{"id": "some-request-id","site": {"page": "prebid.org"},"imp": [{"id": "some-impression-id","banner": {"format": [{"w": 300,"h": 250},{"w": 300,"h": 600}]},"ext": {"appnexus": {"placementId": 10433394}}}],"tmax": 500}`), } - resolvedRequest := json.RawMessage(`{"id": "some-request-id","site": {"page": "prebid.org"},"imp": [{"id": "some-impression-id","banner": {"format": [{"w": 300,"h": 250},{"w": 300,"h": 600}]},"ext": {"appnexus": {"placementId": 1}}}],"tmax": 500}`) - adapterExtra := map[openrtb_ext.BidderName]*seatResponseExtra{ "appnexus": {ResponseTimeMillis: 5}, } @@ -448,9 +974,13 @@ func TestBidResponseCurrency(t *testing.T) { }, } + bidResponseExt := &openrtb_ext.ExtBidResponse{ + ResponseTimeMillis: map[openrtb_ext.BidderName]int{openrtb_ext.BidderName("appnexus"): 5}, + RequestTimeoutMillis: 500, + } // Run tests for i := range testCases { - actualBidResp, err := e.buildBidResponse(context.Background(), liveAdapters, testCases[i].adapterBids, bidRequest, resolvedRequest, adapterExtra, nil, nil, false, errList) + actualBidResp, err := e.buildBidResponse(context.Background(), liveAdapters, testCases[i].adapterBids, bidRequest, adapterExtra, nil, bidResponseExt, true, errList) assert.NoError(t, err, fmt.Sprintf("[TEST_FAILED] e.buildBidResponse resturns error in test: %s Error message: %s \n", testCases[i].description, err)) assert.Equalf(t, testCases[i].expectedBidResponse, actualBidResp, fmt.Sprintf("[TEST_FAILED] Objects must be equal for test: %s \n Expected: >>%s<< \n Actual: >>%s<< ", testCases[i].description, testCases[i].expectedBidResponse.Ext, actualBidResp.Ext)) } @@ -493,8 +1023,16 @@ func TestRaceIntegration(t *testing.T) { t.Errorf("Failed to create a category Fetcher: %v", error) } theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - ex := NewExchange(server.Client(), &wellBehavedCache{}, cfg, theMetrics, adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()) - _, err := ex.HoldAuction(context.Background(), newRaceCheckingRequest(t), &emptyUsersync{}, pbsmetrics.Labels{}, &categoriesFetcher, nil) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + + auctionRequest := AuctionRequest{ + BidRequest: newRaceCheckingRequest(t), + Account: config.Account{}, + UserSyncs: &emptyUsersync{}, + } + + ex := NewExchange(server.Client(), &wellBehavedCache{}, cfg, theMetrics, adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, categoriesFetcher) + _, err := ex.HoldAuction(context.Background(), auctionRequest, nil) if err != nil { t.Errorf("HoldAuction returned unexpected error: %v", err) } @@ -577,7 +1115,8 @@ func TestPanicRecovery(t *testing.T) { } theMetrics := pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}) - e := NewExchange(&http.Client{}, nil, cfg, theMetrics, adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()).(*exchange) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e := NewExchange(&http.Client{}, nil, cfg, theMetrics, adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, nilCategoryFetcher{}).(*exchange) chBids := make(chan *bidResponseWrapper, 1) panicker := func(aName openrtb_ext.BidderName, coreBidder openrtb_ext.BidderName, request *openrtb.BidRequest, bidlabels *pbsmetrics.AdapterLabels, conversions currencies.Conversions) { panic("panic!") @@ -642,7 +1181,12 @@ func TestPanicRecoveryHighLevel(t *testing.T) { Endpoint: server.URL, } } - e := NewExchange(server.Client(), &mockCache{}, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencies.NewRateConverterDefault()).(*exchange) + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + e := NewExchange(server.Client(), &mockCache{}, cfg, pbsmetrics.NewMetrics(metrics.NewRegistry(), openrtb_ext.BidderList(), config.DisabledMetrics{}), adapters.ParseBidderInfos(cfg.Adapters, "../static/bidder-info", openrtb_ext.BidderList()), gdpr.AlwaysAllow{}, currencyConverter, categoriesFetcher).(*exchange) e.adapterMap[openrtb_ext.BidderBeachfront] = panicingAdapter{} e.adapterMap[openrtb_ext.BidderAppnexus] = panicingAdapter{} @@ -675,11 +1219,13 @@ func TestPanicRecoveryHighLevel(t *testing.T) { }}, } - categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") - if error != nil { - t.Errorf("Failed to create a category Fetcher: %v", error) + auctionRequest := AuctionRequest{ + BidRequest: request, + Account: config.Account{}, + UserSyncs: &emptyUsersync{}, } - _, err := e.HoldAuction(context.Background(), request, &emptyUsersync{}, pbsmetrics.Labels{}, &categoriesFetcher, nil) + + _, err := e.HoldAuction(context.Background(), auctionRequest, nil) if err != nil { t.Errorf("HoldAuction returned unexpected error: %v", err) } @@ -703,6 +1249,38 @@ func TestTimeoutComputation(t *testing.T) { } } +func TestSetDebugContextKey(t *testing.T) { + // Test cases + testCases := []struct { + desc string + inDebugInfo bool + expectedDebugInfo bool + }{ + { + desc: "debugInfo flag on, we expect to find DebugContextKey key in context", + inDebugInfo: true, + expectedDebugInfo: true, + }, + { + desc: "debugInfo flag off, we don't expect to find DebugContextKey key in context", + inDebugInfo: false, + expectedDebugInfo: false, + }, + } + + // Setup test + ex := exchange{} + + // Run tests + for _, test := range testCases { + auctionCtx := ex.makeDebugContext(context.Background(), test.inDebugInfo) + + debugInfo := auctionCtx.Value(DebugContextKey) + assert.NotNil(t, debugInfo, "%s. Flag set, `debugInfo` shouldn't be nil") + assert.Equal(t, test.expectedDebugInfo, debugInfo.(bool), "Desc: %s. Incorrect value mapped to DebugContextKey(`debugInfo`) in the context\n", test.desc) + } +} + // TestExchangeJSON executes tests for all the *.json files in exchangetest. func TestExchangeJSON(t *testing.T) { if specFiles, err := ioutil.ReadDir("./exchangetest"); err == nil { @@ -740,6 +1318,12 @@ func runSpec(t *testing.T, filename string, spec *exchangeSpec) { t.Fatalf("%s: Failed to parse aliases", filename) } + var s struct{} + eeac := make(map[string]struct{}) + for _, c := range []string{"FIN", "FRA", "GUF"} { + eeac[c] = s + } + privacyConfig := config.Privacy{ CCPA: config.CCPA{ Enforce: spec.EnforceCCPA, @@ -747,20 +1331,28 @@ func runSpec(t *testing.T, filename string, spec *exchangeSpec) { LMT: config.LMT{ Enforce: spec.EnforceLMT, }, + GDPR: config.GDPR{ + Enabled: spec.GDPREnabled, + UsersyncIfAmbiguous: !spec.AssumeGDPRApplies, + EEACountriesMap: eeac, + }, } ex := newExchangeForTests(t, filename, spec.OutgoingRequests, aliases, privacyConfig) biddersInAuction := findBiddersInAuction(t, filename, &spec.IncomingRequest.OrtbRequest) - categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") - if error != nil { - t.Errorf("Failed to create a category Fetcher: %v", error) - } debugLog := &DebugLog{} if spec.DebugLog != nil { *debugLog = *spec.DebugLog debugLog.Regexp = regexp.MustCompile(`[<>]`) } - bid, err := ex.HoldAuction(context.Background(), &spec.IncomingRequest.OrtbRequest, mockIdFetcher(spec.IncomingRequest.Usersyncs), pbsmetrics.Labels{}, &categoriesFetcher, debugLog) + + auctionRequest := AuctionRequest{ + BidRequest: &spec.IncomingRequest.OrtbRequest, + Account: config.Account{}, + UserSyncs: mockIdFetcher(spec.IncomingRequest.Usersyncs), + } + + bid, err := ex.HoldAuction(context.Background(), auctionRequest, debugLog) responseTimes := extractResponseTimes(t, filename, bid) for _, bidderName := range biddersInAuction { if _, ok := responseTimes[bidderName]; !ok { @@ -864,15 +1456,21 @@ func newExchangeForTests(t *testing.T, filename string, expectations map[string] } } + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Fatalf("Failed to create a category Fetcher: %v", error) + } + return &exchange{ adapterMap: adapters, me: metricsConf.NewMetricsEngine(&config.Configuration{}, openrtb_ext.BidderList()), cache: &wellBehavedCache{}, cacheTime: 0, - gDPR: gdpr.AlwaysAllow{}, - currencyConverter: currencies.NewRateConverterDefault(), - UsersyncIfAmbiguous: false, + gDPR: gdpr.AlwaysFail{}, + currencyConverter: currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)), + UsersyncIfAmbiguous: privacyConfig.GDPR.UsersyncIfAmbiguous, privacyConfig: privacyConfig, + categoriesFetcher: categoriesFetcher, } } @@ -976,7 +1574,7 @@ func TestCategoryMapping(t *testing.T) { adapterBids[bidderName1] = &seatBid - bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, requestExt, adapterBids, categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) assert.Equal(t, nil, err, "Category mapping error should be empty") assert.Equal(t, 1, len(rejections), "There should be 1 bid rejection message") @@ -1032,7 +1630,7 @@ func TestCategoryMappingNoIncludeBrandCategory(t *testing.T) { adapterBids[bidderName1] = &seatBid - bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, requestExt, adapterBids, categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) assert.Equal(t, nil, err, "Category mapping error should be empty") assert.Empty(t, rejections, "There should be no bid rejection messages") @@ -1085,7 +1683,7 @@ func TestCategoryMappingTranslateCategoriesNil(t *testing.T) { adapterBids[bidderName1] = &seatBid - bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, requestExt, adapterBids, categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) assert.Equal(t, nil, err, "Category mapping error should be empty") assert.Equal(t, 1, len(rejections), "There should be 1 bid rejection message") @@ -1168,7 +1766,7 @@ func TestCategoryMappingTranslateCategoriesFalse(t *testing.T) { adapterBids[bidderName1] = &seatBid - bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, requestExt, adapterBids, categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) assert.Equal(t, nil, err, "Category mapping error should be empty") assert.Empty(t, rejections, "There should be no bid rejection messages") @@ -1202,19 +1800,22 @@ func TestCategoryDedupe(t *testing.T) { cats4 := []string{"IAB1-2000"} bid1 := openrtb.Bid{ID: "bid_id1", ImpID: "imp_id1", Price: 10.0000, Cat: cats1, W: 1, H: 1} bid2 := openrtb.Bid{ID: "bid_id2", ImpID: "imp_id2", Price: 15.0000, Cat: cats2, W: 1, H: 1} - bid3 := openrtb.Bid{ID: "bid_id3", ImpID: "imp_id3", Price: 10.0000, Cat: cats1, W: 1, H: 1} + bid3 := openrtb.Bid{ID: "bid_id3", ImpID: "imp_id3", Price: 20.0000, Cat: cats1, W: 1, H: 1} bid4 := openrtb.Bid{ID: "bid_id4", ImpID: "imp_id4", Price: 20.0000, Cat: cats4, W: 1, H: 1} + bid5 := openrtb.Bid{ID: "bid_id5", ImpID: "imp_id5", Price: 20.0000, Cat: cats1, W: 1, H: 1} bid1_1 := pbsOrtbBid{&bid1, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} bid1_2 := pbsOrtbBid{&bid2, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 50}, 0, false} bid1_3 := pbsOrtbBid{&bid3, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} bid1_4 := pbsOrtbBid{&bid4, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_5 := pbsOrtbBid{&bid5, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} selectedBids := make(map[string]int) expectedCategories := map[string]string{ "bid_id1": "10.00_Electronics_30s", "bid_id2": "14.00_Sports_50s", - "bid_id3": "10.00_Electronics_30s", + "bid_id3": "20.00_Electronics_30s", + "bid_id5": "20.00_Electronics_30s", } numIterations := 10 @@ -1228,6 +1829,7 @@ func TestCategoryDedupe(t *testing.T) { &bid1_2, &bid1_3, &bid1_4, + &bid1_5, } seatBid := pbsOrtbSeatBid{innerBids, "USD", nil, nil} @@ -1235,10 +1837,10 @@ func TestCategoryDedupe(t *testing.T) { adapterBids[bidderName1] = &seatBid - bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, requestExt, adapterBids, categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) assert.Equal(t, nil, err, "Category mapping error should be empty") - assert.Equal(t, 2, len(rejections), "There should be 2 bid rejection messages") + assert.Equal(t, 3, len(rejections), "There should be 2 bid rejection messages") assert.Regexpf(t, regexp.MustCompile(`bid rejected \[bid ID: bid_id(1|3)\] reason: Bid was deduplicated`), rejections[0], "Rejection message did not match expected") assert.Equal(t, "bid rejected [bid ID: bid_id4] reason: Category mapping file for primary ad server: 'freewheel', publisher: '' not found", rejections[1], "Rejection message did not match expected") assert.Equal(t, 2, len(adapterBids[bidderName1].bids), "Bidders number doesn't match") @@ -1251,8 +1853,201 @@ func TestCategoryDedupe(t *testing.T) { } assert.Equal(t, numIterations, selectedBids["bid_id2"], "Bid 2 did not make it through every time") - assert.NotEqual(t, numIterations, selectedBids["bid_id1"], "Bid 1 made it through every time") - assert.NotEqual(t, numIterations, selectedBids["bid_id3"], "Bid 3 made it through every time") + assert.Equal(t, 0, selectedBids["bid_id1"], "Bid 1 should be rejected on every iteration due to lower price") + assert.NotEqual(t, 0, selectedBids["bid_id3"], "Bid 3 should be accepted at least once") + assert.NotEqual(t, 0, selectedBids["bid_id5"], "Bid 5 should be accepted at least once") +} + +func TestNoCategoryDedupe(t *testing.T) { + + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + + bidRequest := &openrtb.BidRequest{} + requestExt := newExtRequestNoBrandCat() + + targData := &targetData{ + priceGranularity: requestExt.Prebid.Targeting.PriceGranularity, + includeWinners: true, + } + + adapterBids := make(map[openrtb_ext.BidderName]*pbsOrtbSeatBid) + + cats1 := []string{"IAB1-3"} + cats2 := []string{"IAB1-4"} + cats4 := []string{"IAB1-2000"} + bid1 := openrtb.Bid{ID: "bid_id1", ImpID: "imp_id1", Price: 14.0000, Cat: cats1, W: 1, H: 1} + bid2 := openrtb.Bid{ID: "bid_id2", ImpID: "imp_id2", Price: 14.0000, Cat: cats2, W: 1, H: 1} + bid3 := openrtb.Bid{ID: "bid_id3", ImpID: "imp_id3", Price: 20.0000, Cat: cats1, W: 1, H: 1} + bid4 := openrtb.Bid{ID: "bid_id4", ImpID: "imp_id4", Price: 20.0000, Cat: cats4, W: 1, H: 1} + bid5 := openrtb.Bid{ID: "bid_id5", ImpID: "imp_id5", Price: 10.0000, Cat: cats1, W: 1, H: 1} + + bid1_1 := pbsOrtbBid{&bid1, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_2 := pbsOrtbBid{&bid2, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_3 := pbsOrtbBid{&bid3, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_4 := pbsOrtbBid{&bid4, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_5 := pbsOrtbBid{&bid5, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + + selectedBids := make(map[string]int) + expectedCategories := map[string]string{ + "bid_id1": "14.00_30s", + "bid_id2": "14.00_30s", + "bid_id3": "20.00_30s", + "bid_id4": "20.00_30s", + "bid_id5": "10.00_30s", + } + + numIterations := 10 + + // Run the function many times, this should be enough for the 50% chance of which bid to remove to remove bid1 sometimes + // and bid3 others. It's conceivably possible (but highly unlikely) that the same bid get chosen every single time, but + // if you notice false fails from this test increase numIterations to make it even less likely to happen. + for i := 0; i < numIterations; i++ { + innerBids := []*pbsOrtbBid{ + &bid1_1, + &bid1_2, + &bid1_3, + &bid1_4, + &bid1_5, + } + + seatBid := pbsOrtbSeatBid{innerBids, "USD", nil, nil} + bidderName1 := openrtb_ext.BidderName("appnexus") + + adapterBids[bidderName1] = &seatBid + + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) + + assert.Equal(t, nil, err, "Category mapping error should be empty") + assert.Equal(t, 2, len(rejections), "There should be 2 bid rejection messages") + assert.Regexpf(t, regexp.MustCompile(`bid rejected \[bid ID: bid_id(1|2)\] reason: Bid was deduplicated`), rejections[0], "Rejection message did not match expected") + assert.Regexpf(t, regexp.MustCompile(`bid rejected \[bid ID: bid_id(3|4)\] reason: Bid was deduplicated`), rejections[1], "Rejection message did not match expected") + assert.Equal(t, 3, len(adapterBids[bidderName1].bids), "Bidders number doesn't match") + assert.Equal(t, 3, len(bidCategory), "Bidders category mapping doesn't match") + + for bidId, bidCat := range bidCategory { + assert.Equal(t, expectedCategories[bidId], bidCat, "Category mapping doesn't match") + selectedBids[bidId]++ + } + } + assert.Equal(t, numIterations, selectedBids["bid_id5"], "Bid 5 did not make it through every time") + assert.NotEqual(t, 0, selectedBids["bid_id1"], "Bid 1 should be selected at least once") + assert.NotEqual(t, 0, selectedBids["bid_id2"], "Bid 2 should be selected at least once") + assert.NotEqual(t, 0, selectedBids["bid_id1"], "Bid 3 should be selected at least once") + assert.NotEqual(t, 0, selectedBids["bid_id4"], "Bid 4 should be selected at least once") + +} + +func TestCategoryMappingBidderName(t *testing.T) { + + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + + bidRequest := &openrtb.BidRequest{} + requestExt := newExtRequest() + requestExt.Prebid.Targeting.AppendBidderNames = true + + targData := &targetData{ + priceGranularity: requestExt.Prebid.Targeting.PriceGranularity, + includeWinners: true, + } + + requestExt.Prebid.Targeting.DurationRangeSec = []int{15, 30} + + adapterBids := make(map[openrtb_ext.BidderName]*pbsOrtbSeatBid) + + cats1 := []string{"IAB1-1"} + cats2 := []string{"IAB1-2"} + bid1 := openrtb.Bid{ID: "bid_id1", ImpID: "imp_id1", Price: 10.0000, Cat: cats1, W: 1, H: 1} + bid2 := openrtb.Bid{ID: "bid_id2", ImpID: "imp_id2", Price: 10.0000, Cat: cats2, W: 1, H: 1} + + bid1_1 := pbsOrtbBid{&bid1, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_2 := pbsOrtbBid{&bid2, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + + innerBids1 := []*pbsOrtbBid{ + &bid1_1, + } + innerBids2 := []*pbsOrtbBid{ + &bid1_2, + } + + seatBid1 := pbsOrtbSeatBid{innerBids1, "USD", nil, nil} + bidderName1 := openrtb_ext.BidderName("bidder1") + + seatBid2 := pbsOrtbSeatBid{innerBids2, "USD", nil, nil} + bidderName2 := openrtb_ext.BidderName("bidder2") + + adapterBids[bidderName1] = &seatBid1 + adapterBids[bidderName2] = &seatBid2 + + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) + + assert.NoError(t, err, "Category mapping error should be empty") + assert.Empty(t, rejections, "There should be 0 bid rejection messages") + assert.Equal(t, "10.00_VideoGames_30s_bidder1", bidCategory["bid_id1"], "Category mapping doesn't match") + assert.Equal(t, "10.00_HomeDecor_30s_bidder2", bidCategory["bid_id2"], "Category mapping doesn't match") + assert.Len(t, adapterBids[bidderName1].bids, 1, "Bidders number doesn't match") + assert.Len(t, adapterBids[bidderName2].bids, 1, "Bidders number doesn't match") + assert.Len(t, bidCategory, 2, "Bidders category mapping doesn't match") +} + +func TestCategoryMappingBidderNameNoCategories(t *testing.T) { + + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + + bidRequest := &openrtb.BidRequest{} + requestExt := newExtRequestNoBrandCat() + requestExt.Prebid.Targeting.AppendBidderNames = true + + targData := &targetData{ + priceGranularity: requestExt.Prebid.Targeting.PriceGranularity, + includeWinners: true, + } + + requestExt.Prebid.Targeting.DurationRangeSec = []int{15, 30} + + adapterBids := make(map[openrtb_ext.BidderName]*pbsOrtbSeatBid) + + cats1 := []string{"IAB1-1"} + cats2 := []string{"IAB1-2"} + bid1 := openrtb.Bid{ID: "bid_id1", ImpID: "imp_id1", Price: 10.0000, Cat: cats1, W: 1, H: 1} + bid2 := openrtb.Bid{ID: "bid_id2", ImpID: "imp_id2", Price: 12.0000, Cat: cats2, W: 1, H: 1} + + bid1_1 := pbsOrtbBid{&bid1, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_2 := pbsOrtbBid{&bid2, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + + innerBids1 := []*pbsOrtbBid{ + &bid1_1, + } + innerBids2 := []*pbsOrtbBid{ + &bid1_2, + } + + seatBid1 := pbsOrtbSeatBid{innerBids1, "USD", nil, nil} + bidderName1 := openrtb_ext.BidderName("bidder1") + + seatBid2 := pbsOrtbSeatBid{innerBids2, "USD", nil, nil} + bidderName2 := openrtb_ext.BidderName("bidder2") + + adapterBids[bidderName1] = &seatBid1 + adapterBids[bidderName2] = &seatBid2 + + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) + + assert.NoError(t, err, "Category mapping error should be empty") + assert.Empty(t, rejections, "There should be 0 bid rejection messages") + assert.Equal(t, "10.00_30s_bidder1", bidCategory["bid_id1"], "Category mapping doesn't match") + assert.Equal(t, "12.00_30s_bidder2", bidCategory["bid_id2"], "Category mapping doesn't match") + assert.Len(t, adapterBids[bidderName1].bids, 1, "Bidders number doesn't match") + assert.Len(t, adapterBids[bidderName2].bids, 1, "Bidders number doesn't match") + assert.Len(t, bidCategory, 2, "Bidders category mapping doesn't match") } func TestBidRejectionErrors(t *testing.T) { @@ -1347,7 +2142,7 @@ func TestBidRejectionErrors(t *testing.T) { adapterBids[bidderName] = &seatBid - bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, test.reqExt, adapterBids, categoriesFetcher, targData) + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &test.reqExt, adapterBids, categoriesFetcher, targData) if len(test.expectedCatDur) > 0 { // Bid deduplication case @@ -1364,6 +2159,79 @@ func TestBidRejectionErrors(t *testing.T) { } } +func TestCategoryMappingTwoBiddersOneBidEachNoCategorySamePrice(t *testing.T) { + + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + + bidRequest := &openrtb.BidRequest{} + requestExt := newExtRequestTranslateCategories(nil) + + targData := &targetData{ + priceGranularity: requestExt.Prebid.Targeting.PriceGranularity, + includeWinners: true, + } + + requestExt.Prebid.Targeting.DurationRangeSec = []int{30} + requestExt.Prebid.Targeting.IncludeBrandCategory.WithCategory = false + + cats1 := []string{"IAB1-3"} + cats2 := []string{"IAB1-4"} + + bidApn1 := openrtb.Bid{ID: "bid_idApn1", ImpID: "imp_idApn1", Price: 10.0000, Cat: cats1, W: 1, H: 1} + bidApn2 := openrtb.Bid{ID: "bid_idApn2", ImpID: "imp_idApn2", Price: 10.0000, Cat: cats2, W: 1, H: 1} + + bid1_Apn1 := pbsOrtbBid{&bidApn1, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + bid1_Apn2 := pbsOrtbBid{&bidApn2, "video", nil, &openrtb_ext.ExtBidPrebidVideo{Duration: 30}, 0, false} + + innerBidsApn1 := []*pbsOrtbBid{ + &bid1_Apn1, + } + + innerBidsApn2 := []*pbsOrtbBid{ + &bid1_Apn2, + } + + for i := 1; i < 10; i++ { + adapterBids := make(map[openrtb_ext.BidderName]*pbsOrtbSeatBid) + + seatBidApn1 := pbsOrtbSeatBid{innerBidsApn1, "USD", nil, nil} + bidderNameApn1 := openrtb_ext.BidderName("appnexus1") + + seatBidApn2 := pbsOrtbSeatBid{innerBidsApn2, "USD", nil, nil} + bidderNameApn2 := openrtb_ext.BidderName("appnexus2") + + adapterBids[bidderNameApn1] = &seatBidApn1 + adapterBids[bidderNameApn2] = &seatBidApn2 + + bidCategory, adapterBids, rejections, err := applyCategoryMapping(nil, bidRequest, &requestExt, adapterBids, categoriesFetcher, targData) + + assert.NoError(t, err, "Category mapping error should be empty") + assert.Len(t, rejections, 1, "There should be 1 bid rejection message") + assert.Regexpf(t, regexp.MustCompile(`bid rejected \[bid ID: bid_idApn(1|2)\] reason: Bid was deduplicated`), rejections[0], "Rejection message did not match expected") + assert.Len(t, bidCategory, 1, "Bidders category mapping should have only one element") + + var resultBid string + for bidId := range bidCategory { + resultBid = bidId + } + + if resultBid == "bid_idApn1" { + assert.Nil(t, seatBidApn2.bids, "Appnexus_2 seat bid should not have any bids back") + assert.Len(t, seatBidApn1.bids, 1, "Appnexus_1 seat bid should have only one back") + + } else { + assert.Nil(t, seatBidApn1.bids, "Appnexus_1 seat bid should not have any bids back") + assert.Len(t, seatBidApn2.bids, 1, "Appnexus_2 seat bid should have only one back") + + } + + } + +} + func TestUpdateRejections(t *testing.T) { rejections := []string{} @@ -1377,12 +2245,13 @@ func TestUpdateRejections(t *testing.T) { func TestApplyDealSupport(t *testing.T) { testCases := []struct { - description string - dealPriority int - impExt json.RawMessage - targ map[string]string - expectedHbPbCatDur string - expectedDealErr string + description string + dealPriority int + impExt json.RawMessage + targ map[string]string + expectedHbPbCatDur string + expectedDealErr string + expectedDealTierSatisfied bool }{ { description: "hb_pb_cat_dur should be modified", @@ -1391,8 +2260,9 @@ func TestApplyDealSupport(t *testing.T) { targ: map[string]string{ "hb_pb_cat_dur": "12.00_movies_30s", }, - expectedHbPbCatDur: "tier5_movies_30s", - expectedDealErr: "", + expectedHbPbCatDur: "tier5_movies_30s", + expectedDealErr: "", + expectedDealTierSatisfied: true, }, { description: "hb_pb_cat_dur should not be modified due to priority not exceeding min", @@ -1401,8 +2271,9 @@ func TestApplyDealSupport(t *testing.T) { targ: map[string]string{ "hb_pb_cat_dur": "12.00_medicine_30s", }, - expectedHbPbCatDur: "12.00_medicine_30s", - expectedDealErr: "", + expectedHbPbCatDur: "12.00_medicine_30s", + expectedDealErr: "", + expectedDealTierSatisfied: false, }, { description: "hb_pb_cat_dur should not be modified due to invalid config", @@ -1411,8 +2282,9 @@ func TestApplyDealSupport(t *testing.T) { targ: map[string]string{ "hb_pb_cat_dur": "12.00_games_30s", }, - expectedHbPbCatDur: "12.00_games_30s", - expectedDealErr: "dealTier configuration invalid for bidder 'appnexus', imp ID 'imp_id1'", + expectedHbPbCatDur: "12.00_games_30s", + expectedDealErr: "dealTier configuration invalid for bidder 'appnexus', imp ID 'imp_id1'", + expectedDealTierSatisfied: false, }, { description: "hb_pb_cat_dur should not be modified due to deal priority of 0", @@ -1421,8 +2293,9 @@ func TestApplyDealSupport(t *testing.T) { targ: map[string]string{ "hb_pb_cat_dur": "12.00_auto_30s", }, - expectedHbPbCatDur: "12.00_auto_30s", - expectedDealErr: "", + expectedHbPbCatDur: "12.00_auto_30s", + expectedDealErr: "", + expectedDealTierSatisfied: false, }, } @@ -1454,6 +2327,7 @@ func TestApplyDealSupport(t *testing.T) { dealErrs := applyDealSupport(bidRequest, auc, bidCategory) assert.Equal(t, test.expectedHbPbCatDur, bidCategory[auc.winningBidsByBidder["imp_id1"][bidderName].bid.ID], test.description) + assert.Equal(t, test.expectedDealTierSatisfied, auc.winningBidsByBidder["imp_id1"][bidderName].dealTierSatisfied, "expectedDealTierSatisfied=%v when %v", test.expectedDealTierSatisfied, test.description) if len(test.expectedDealErr) > 0 { assert.Containsf(t, dealErrs, errors.New(test.expectedDealErr), "Expected error message not found in deal errors") } @@ -1462,129 +2336,102 @@ func TestApplyDealSupport(t *testing.T) { func TestGetDealTiers(t *testing.T) { testCases := []struct { - impExt json.RawMessage - bidderResult map[string]bool // true indicates bidder had valid config, false indicates invalid + description string + request openrtb.BidRequest + expected map[string]openrtb_ext.DealTierBidderMap }{ { - impExt: json.RawMessage(`{"validbase": {"dealTier": {"minDealTier": 5, "prefix": "tier"}, "placementId": 10433394}}`), - bidderResult: map[string]bool{ - "validbase": true, + description: "None", + request: openrtb.BidRequest{ + Imp: []openrtb.Imp{}, }, + expected: map[string]openrtb_ext.DealTierBidderMap{}, }, { - impExt: json.RawMessage(`{"validmultiple1": {"dealTier": {"minDealTier": 5, "prefix": "tier"}, "placementId": 10433394}, "validmultiple2": {"dealTier": {"minDealTier": 5, "prefix": "tier"}, "placementId": 10433394}}`), - bidderResult: map[string]bool{ - "validmultiple1": true, - "validmultiple2": true, + description: "One", + request: openrtb.BidRequest{ + Imp: []openrtb.Imp{ + {ID: "imp1", Ext: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "tier"}}}`)}, + }, + }, + expected: map[string]openrtb_ext.DealTierBidderMap{ + "imp1": {openrtb_ext.BidderAppnexus: {Prefix: "tier", MinDealTier: 5}}, }, }, { - impExt: json.RawMessage(`{"nodealtier": {"placementId": 10433394}}`), - bidderResult: map[string]bool{ - "nodealtier": false, + description: "Many", + request: openrtb.BidRequest{ + Imp: []openrtb.Imp{ + {ID: "imp1", Ext: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "tier1"}}}`)}, + {ID: "imp2", Ext: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 8, "prefix": "tier2"}}}`)}, + }, + }, + expected: map[string]openrtb_ext.DealTierBidderMap{ + "imp1": {openrtb_ext.BidderAppnexus: {Prefix: "tier1", MinDealTier: 5}}, + "imp2": {openrtb_ext.BidderAppnexus: {Prefix: "tier2", MinDealTier: 8}}, }, }, { - impExt: json.RawMessage(`{"validbase": {"placementId": 10433394}, "onedealTier2": {"dealTier": {"minDealTier": 5, "prefix": "tier"}, "placementId": 10433394}}`), - bidderResult: map[string]bool{ - "onedealTier2": true, - "validbase": false, + description: "Many - Skips Malformed", + request: openrtb.BidRequest{ + Imp: []openrtb.Imp{ + {ID: "imp1", Ext: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "tier1"}}}`)}, + {ID: "imp2", Ext: json.RawMessage(`{"appnexus": {"dealTier": "wrong type"}}`)}, + }, + }, + expected: map[string]openrtb_ext.DealTierBidderMap{ + "imp1": {openrtb_ext.BidderAppnexus: {Prefix: "tier1", MinDealTier: 5}}, }, }, } - filledDealTier := DealTier{ - Info: &DealTierInfo{ - Prefix: "tier", - MinDealTier: 5, - }, - } - emptyDealTier := DealTier{} - for _, test := range testCases { - bidRequest := &openrtb.BidRequest{ - ID: "some-request-id", - Imp: []openrtb.Imp{ - { - ID: "imp_id1", - Ext: test.impExt, - }, - }, - } - - impDealMap := getDealTiers(bidRequest) - - for bidder, valid := range test.bidderResult { - if valid { - assert.Equal(t, &filledDealTier, impDealMap["imp_id1"].DealInfo[bidder], "DealTier should be filled with config data") - } else { - assert.Equal(t, &emptyDealTier, impDealMap["imp_id1"].DealInfo[bidder], "DealTier should be empty") - } - } + result := getDealTiers(&test.request) + assert.Equal(t, test.expected, result, test.description) } } -func TestValidateAndNormalizeDealTier(t *testing.T) { +func TestValidateDealTier(t *testing.T) { testCases := []struct { description string - params json.RawMessage + dealTier openrtb_ext.DealTier expectedResult bool }{ { - description: "BidderDealTier should be valid", - params: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "tier"}, "placementId": 10433394}}`), + description: "Valid", + dealTier: openrtb_ext.DealTier{Prefix: "prefix", MinDealTier: 5}, expectedResult: true, }, { - description: "BidderDealTier should be invalid due to empty prefix", - params: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": ""}, "placementId": 10433394}}`), - expectedResult: false, - }, - { - description: "BidderDealTier should be invalid due to empty dealTier", - params: json.RawMessage(`{"appnexus": {"dealTier": {}, "placementId": 10433394}}`), - expectedResult: false, - }, - { - description: "BidderDealTier should be invalid due to missing minDealTier", - params: json.RawMessage(`{"appnexus": {"dealTier": {"prefix": "tier"}, "placementId": 10433394}}`), + description: "Invalid - Empty", + dealTier: openrtb_ext.DealTier{}, expectedResult: false, }, { - description: "BidderDealTier should be invalid due to missing dealTier", - params: json.RawMessage(`{"appnexus": {"placementId": 10433394}}`), + description: "Invalid - Empty Prefix", + dealTier: openrtb_ext.DealTier{MinDealTier: 5}, expectedResult: false, }, { - description: "BidderDealTier should be invalid due to prefix containing all whitespace", - params: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": " "}, "placementId": 10433394}}`), + description: "Invalid - Empty Deal Tier", + dealTier: openrtb_ext.DealTier{Prefix: "prefix"}, expectedResult: false, }, - { - description: "BidderDealTier should be valid after removing whitespace", - params: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": " prefixwith sp aces "}, "placementId": 10433394}}`), - expectedResult: true, - }, } for _, test := range testCases { - var bidderDealTier BidderDealTier - err := json.Unmarshal(test.params, &bidderDealTier.DealInfo) - if err != nil { - assert.Fail(t, "Unable to unmarshal JSON data for testing BidderDealTier") - } - - assert.Equal(t, test.expectedResult, validateAndNormalizeDealTier(bidderDealTier.DealInfo["appnexus"]), test.description) + assert.Equal(t, test.expectedResult, validateDealTier(test.dealTier), test.description) } } func TestUpdateHbPbCatDur(t *testing.T) { testCases := []struct { - description string - targ map[string]string - dealTier *DealTierInfo - dealPriority int - expectedHbPbCatDur string + description string + targ map[string]string + dealTier openrtb_ext.DealTier + dealPriority int + expectedHbPbCatDur string + expectedDealTierSatisfied bool }{ { description: "hb_pb_cat_dur should be updated with prefix and tier", @@ -1592,12 +2439,13 @@ func TestUpdateHbPbCatDur(t *testing.T) { "hb_pb": "12.00", "hb_pb_cat_dur": "12.00_movies_30s", }, - dealTier: &DealTierInfo{ + dealTier: openrtb_ext.DealTier{ Prefix: "tier", MinDealTier: 5, }, - dealPriority: 5, - expectedHbPbCatDur: "tier5_movies_30s", + dealPriority: 5, + expectedHbPbCatDur: "tier5_movies_30s", + expectedDealTierSatisfied: true, }, { description: "hb_pb_cat_dur should not be updated due to bid priority", @@ -1605,12 +2453,13 @@ func TestUpdateHbPbCatDur(t *testing.T) { "hb_pb": "12.00", "hb_pb_cat_dur": "12.00_auto_30s", }, - dealTier: &DealTierInfo{ + dealTier: openrtb_ext.DealTier{ Prefix: "tier", MinDealTier: 10, }, - dealPriority: 6, - expectedHbPbCatDur: "12.00_auto_30s", + dealPriority: 6, + expectedHbPbCatDur: "12.00_auto_30s", + expectedDealTierSatisfied: false, }, { description: "hb_pb_cat_dur should be updated with prefix and tier", @@ -1618,12 +2467,13 @@ func TestUpdateHbPbCatDur(t *testing.T) { "hb_pb": "12.00", "hb_pb_cat_dur": "12.00_medicine_30s", }, - dealTier: &DealTierInfo{ + dealTier: openrtb_ext.DealTier{ Prefix: "tier", MinDealTier: 1, }, - dealPriority: 7, - expectedHbPbCatDur: "tier7_medicine_30s", + dealPriority: 7, + expectedHbPbCatDur: "tier7_medicine_30s", + expectedDealTierSatisfied: true, }, } @@ -1636,6 +2486,7 @@ func TestUpdateHbPbCatDur(t *testing.T) { updateHbPbCatDur(&bid, test.dealTier, bidCategory) assert.Equal(t, test.expectedHbPbCatDur, bidCategory[bid.bid.ID], test.description) + assert.Equal(t, test.expectedDealTierSatisfied, bid.dealTierSatisfied, test.description) } } @@ -1684,12 +2535,14 @@ func TestRecordAdaptorDuplicateBidIDs(t *testing.T) { } type exchangeSpec struct { - IncomingRequest exchangeRequest `json:"incomingRequest"` - OutgoingRequests map[string]*bidderSpec `json:"outgoingRequests"` - Response exchangeResponse `json:"response,omitempty"` - EnforceCCPA bool `json:"enforceCcpa"` - EnforceLMT bool `json:"enforceLmt"` - DebugLog *DebugLog `json:"debuglog,omitempty"` + GDPREnabled bool `json:"gdpr_enabled"` + IncomingRequest exchangeRequest `json:"incomingRequest"` + OutgoingRequests map[string]*bidderSpec `json:"outgoingRequests"` + Response exchangeResponse `json:"response,omitempty"` + EnforceCCPA bool `json:"enforceCcpa"` + EnforceLMT bool `json:"enforceLmt"` + AssumeGDPRApplies bool `json:"assume_gdpr_applies"` + DebugLog *DebugLog `json:"debuglog,omitempty"` } type exchangeRequest struct { @@ -1740,6 +2593,10 @@ func (f mockIdFetcher) GetId(bidder openrtb_ext.BidderName) (id string, ok bool) return } +func (f mockIdFetcher) LiveSyncCount() int { + return len(f) +} + type validatingBidder struct { t *testing.T fileName string @@ -1750,7 +2607,7 @@ type validatingBidder struct { mockResponses map[string]bidderResponse } -func (b *validatingBidder) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (seatBid *pbsOrtbSeatBid, errs []error) { +func (b *validatingBidder) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (seatBid *pbsOrtbSeatBid, errs []error) { if expectedRequest, ok := b.expectations[string(name)]; ok { if expectedRequest != nil { if expectedRequest.BidAdjustment != bidAdjustment { @@ -1881,13 +2738,22 @@ func mockHandler(statusCode int, getBody string, postBody string) http.Handler { }) } +func mockSlowHandler(delay time.Duration, statusCode int, body string) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + time.Sleep(delay) + + w.WriteHeader(statusCode) + w.Write([]byte(body)) + }) +} + type wellBehavedCache struct{} -func (c *wellBehavedCache) GetExtCacheData() (string, string) { - return "www.pbcserver.com", "/pbcache/endpoint" +func (c *wellBehavedCache) GetExtCacheData() (scheme string, host string, path string) { + return "https", "www.pbcserver.com", "/pbcache/endpoint" } -func (c *wellBehavedCache) PutJson(ctx context.Context, values []prebid_cache_client.Cacheable) ([]string, []error) { +func (c *wellBehavedCache) PutJson(ctx context.Context, values []pbc.Cacheable) ([]string, []error) { ids := make([]string, len(values)) for i := 0; i < len(values); i++ { ids[i] = strconv.Itoa(i) @@ -1901,6 +2767,10 @@ func (e *emptyUsersync) GetId(bidder openrtb_ext.BidderName) (string, bool) { return "", false } +func (e *emptyUsersync) LiveSyncCount() int { + return 0 +} + type mockUsersync struct { syncs map[string]string } @@ -1910,8 +2780,18 @@ func (e *mockUsersync) GetId(bidder openrtb_ext.BidderName) (id string, exists b return } +func (e *mockUsersync) LiveSyncCount() int { + return len(e.syncs) +} + type panicingAdapter struct{} -func (panicingAdapter) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (posb *pbsOrtbSeatBid, errs []error) { +func (panicingAdapter) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (posb *pbsOrtbSeatBid, errs []error) { panic("Panic! Panic! The world is ending!") } + +type nilCategoryFetcher struct{} + +func (nilCategoryFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { + return "", nil +} diff --git a/exchange/exchangetest/append-bidder-names.json b/exchange/exchangetest/append-bidder-names.json new file mode 100644 index 00000000000..1247b9f0261 --- /dev/null +++ b/exchange/exchangetest/append-bidder-names.json @@ -0,0 +1,222 @@ +{ + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }, + { + "id": "imp-id-2", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + } + ], + "test": 1, + "ext": { + "prebid": { + "targeting": { + "includebrandcategory": { + "primaryadserver": 1, + "publisher": "", + "withcategory": true + }, + "appendbiddernames": true + } + } + } + }, + "usersyncs": { + "appnexus": "123" + } + }, + "outgoingRequests": { + "appnexus": { + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [ + { + "ortbBid": { + "id": "apn-bid", + "impid": "my-imp-id", + "price": 0.3, + "w": 200, + "h": 250, + "crid": "creative-1", + "cat": [ + "IAB1-1" + ] + }, + "bidType": "video", + "bidVideo": { + "duration": 30, + "PrimaryCategory": "" + } + }, + { + "ortbBid": { + "id": "apn-other-bid", + "impid": "imp-id-2", + "price": 0.6, + "w": 300, + "h": 500, + "crid": "creative-3", + "cat": [ + "IAB1-2" + ] + }, + "bidType": "video" + } + ] + } + } + } + }, + "response": { + "bids": { + "id": "some-request-id", + "seatbid": [ + { + "seat": "appnexus", + "bid": [ + { + "id": "apn-bid", + "impid": "my-imp-id", + "price": 0.3, + "w": 200, + "h": 250, + "crid": "creative-1", + "cat": [ + "IAB1-1" + ], + "ext": { + "prebid": { + "type": "video", + "targeting": { + "hb_bidder": "appnexus", + "hb_bidder_appnexus": "appnexus", + "hb_cache_host": "www.pbcserver.com", + "hb_cache_host_appnex": "www.pbcserver.com", + "hb_cache_path": "/pbcache/endpoint", + "hb_cache_path_appnex": "/pbcache/endpoint", + "hb_pb": "0.20", + "hb_pb_appnexus": "0.20", + "hb_pb_cat_dur": "0.20_VideoGames_0s_appnexus", + "hb_pb_cat_dur_appnex": "0.20_VideoGames_0s_appnexus", + "hb_size": "200x250", + "hb_size_appnexus": "200x250" + } + } + } + }, + { + "cat": [ + "IAB1-2" + ], + "crid": "creative-3", + "ext": { + "prebid": { + "targeting": { + "hb_bidder": "appnexus", + "hb_bidder_appnexus": "appnexus", + "hb_cache_host": "www.pbcserver.com", + "hb_cache_host_appnex": "www.pbcserver.com", + "hb_cache_path": "/pbcache/endpoint", + "hb_cache_path_appnex": "/pbcache/endpoint", + "hb_pb": "0.50", + "hb_pb_appnexus": "0.50", + "hb_pb_cat_dur": "0.50_HomeDecor_0s_appnexus", + "hb_pb_cat_dur_appnex": "0.50_HomeDecor_0s_appnexus", + "hb_size": "300x500", + "hb_size_appnexus": "300x500" + }, + "type": "video" + } + }, + "h": 500, + "id": "apn-other-bid", + "impid": "imp-id-2", + "price": 0.6, + "w": 300 + } + ] + } + ] + }, + "ext": { + "debug": { + "httpcalls": { + "appnexus": null + }, + "resolvedrequest": { + "id": "some-request-id", + "imp": [ + { + "id": "my-imp-id", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }, + { + "id": "imp-id-2", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + } + ], + "site": { + "page": "test.somepage.com" + }, + "test": 1, + "ext": { + "prebid": { + "targeting": { + "includebrandcategory": { + "primaryadserver": 1, + "publisher": "", + "withcategory": true + }, + "appendbiddernames": true + } + } + } + } + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/ccpa-nosale-any-bidder.json b/exchange/exchangetest/ccpa-nosale-any-bidder.json new file mode 100644 index 00000000000..f7abd91f512 --- /dev/null +++ b/exchange/exchangetest/ccpa-nosale-any-bidder.json @@ -0,0 +1,75 @@ +{ + "enforceCcpa": true, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "regs": { + "ext": { + "us_privacy": "1-Y-" + } + }, + "ext": { + "prebid": { + "nosale": ["*"] + } + }, + "user": { + "buyeruid": "some-buyer-id" + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "regs": { + "ext": { + "us_privacy": "1-Y-" + } + }, + "ext": { + "prebid": { + "nosale": ["*"] + } + }, + "user": { + "buyeruid": "some-buyer-id" + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/ccpa-nosale-specific-bidder.json b/exchange/exchangetest/ccpa-nosale-specific-bidder.json new file mode 100644 index 00000000000..b89e29aea01 --- /dev/null +++ b/exchange/exchangetest/ccpa-nosale-specific-bidder.json @@ -0,0 +1,75 @@ +{ + "enforceCcpa": true, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "regs": { + "ext": { + "us_privacy": "1-Y-" + } + }, + "ext": { + "prebid": { + "nosale": ["appnexus"] + } + }, + "user": { + "buyeruid": "some-buyer-id" + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "regs": { + "ext": { + "us_privacy": "1-Y-" + } + }, + "ext": { + "prebid": { + "nosale": ["appnexus"] + } + }, + "user": { + "buyeruid": "some-buyer-id" + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/debuglog_disabled.json b/exchange/exchangetest/debuglog_disabled.json index 9902dea4bbc..0497e7e2b07 100644 --- a/exchange/exchangetest/debuglog_disabled.json +++ b/exchange/exchangetest/debuglog_disabled.json @@ -46,7 +46,6 @@ "test": 1, "ext": { "prebid": { - "debug" :1, "targeting": { "includebrandcategory": { "primaryadserver": 1, @@ -215,7 +214,6 @@ "test": 1, "ext": { "prebid": { - "debug": 1, "targeting": { "includebrandcategory": { "primaryadserver": 1, @@ -229,4 +227,4 @@ } } } -} \ No newline at end of file +} diff --git a/exchange/exchangetest/debuglog_enabled.json b/exchange/exchangetest/debuglog_enabled.json index 3b307b67e55..885b8b544b3 100644 --- a/exchange/exchangetest/debuglog_enabled.json +++ b/exchange/exchangetest/debuglog_enabled.json @@ -46,7 +46,6 @@ "test": 1, "ext": { "prebid": { - "debug":1, "targeting": { "includebrandcategory": { "primaryadserver": 1, @@ -215,7 +214,6 @@ "test": 1, "ext": { "prebid": { - "debug":1, "targeting": { "includebrandcategory": { "primaryadserver": 1, @@ -229,4 +227,4 @@ } } } -} \ No newline at end of file +} diff --git a/exchange/exchangetest/debuglog_enabled_no_bids.json b/exchange/exchangetest/debuglog_enabled_no_bids.json new file mode 100644 index 00000000000..4823acf8f16 --- /dev/null +++ b/exchange/exchangetest/debuglog_enabled_no_bids.json @@ -0,0 +1,72 @@ +{ + "debugLog": { + "Enabled": true, + "CacheType": "xml", + "TTL": 3600, + "Data": { + "Request": "test request string", + "Headers": "test headers string", + "Response": "" + } + }, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [ + { + "id": "my-imp-id", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }, + { + "id": "imp-id-2", + "video": { + "mimes": [ + "video/mp4" + ] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + } + ], + "ext": { + "prebid": { + "targeting": { + "includebrandcategory": { + "primaryadserver": 1, + "publisher": "", + "withcategory": true + } + } + } + } + }, + "usersyncs": { + "appnexus": "123" + } + }, + "outgoingRequests": { + "appnexus": { + "mockResponse": { + "pbsSeatBid": {} + } + } + }, + "response": { + "bids": {} + } +} \ No newline at end of file diff --git a/exchange/exchangetest/firstpartydata-imp-ext-multiple-bidders.json b/exchange/exchangetest/firstpartydata-imp-ext-multiple-bidders.json new file mode 100644 index 00000000000..8004c3c2646 --- /dev/null +++ b/exchange/exchangetest/firstpartydata-imp-ext-multiple-bidders.json @@ -0,0 +1,173 @@ +{ + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "appnexus": { + "placementId": 1 + }, + "rubicon": { + "accountId": 1, + "siteId": 2, + "zoneId": 3 + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "bidder": { + "placementId": 1 + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [{ + "ortbBid": { + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1" + }, + "bidType": "banner" + }] + } + } + }, + "rubicon": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "bidder": { + "accountId": 1, + "siteId": 2, + "zoneId": 3 + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [{ + "ortbBid": { + "id": "rubi-bid", + "impid": "some-imp-id", + "price": 0.4, + "w": 200, + "h": 500, + "crid": "creative-2" + }, + "bidType": "banner" + }] + } + } + } + }, + "response": { + "bids": { + "id": "some-request-id", + "seatbid": [{ + "seat": "appnexus", + "bid": [{ + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1", + "ext": { + "prebid": { + "type": "banner" + } + } + }] + }, { + "seat": "rubicon", + "bid": [{ + "id": "rubi-bid", + "impid": "some-imp-id", + "price": 0.4, + "w": 200, + "h": 500, + "crid": "creative-2", + "ext": { + "prebid": { + "type": "banner" + } + } + }] + }] + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/firstpartydata-imp-ext-multiple-prebid-bidders.json b/exchange/exchangetest/firstpartydata-imp-ext-multiple-prebid-bidders.json new file mode 100644 index 00000000000..d62afccf426 --- /dev/null +++ b/exchange/exchangetest/firstpartydata-imp-ext-multiple-prebid-bidders.json @@ -0,0 +1,179 @@ +{ + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "prebid": { + "bidder": { + "appnexus": { + "placementId": 1 + }, + "rubicon": { + "accountId": 1, + "siteId": 2, + "zoneId": 3 + } + } + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "bidder": { + "placementId": 1 + }, + "prebid": {}, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [{ + "ortbBid": { + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1" + }, + "bidType": "banner" + }] + } + } + }, + "rubicon": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "bidder": { + "accountId": 1, + "siteId": 2, + "zoneId": 3 + }, + "prebid": {}, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [{ + "ortbBid": { + "id": "rubi-bid", + "impid": "some-imp-id", + "price": 0.4, + "w": 200, + "h": 500, + "crid": "creative-2" + }, + "bidType": "banner" + }] + } + } + } + }, + "response": { + "bids": { + "id": "some-request-id", + "seatbid": [{ + "seat": "appnexus", + "bid": [{ + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1", + "ext": { + "prebid": { + "type": "banner" + } + } + }] + }, { + "seat": "rubicon", + "bid": [{ + "id": "rubi-bid", + "impid": "some-imp-id", + "price": 0.4, + "w": 200, + "h": 500, + "crid": "creative-2", + "ext": { + "prebid": { + "type": "banner" + } + } + }] + }] + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/firstpartydata-imp-ext-one-bidder.json b/exchange/exchangetest/firstpartydata-imp-ext-one-bidder.json new file mode 100644 index 00000000000..6f0bab9529c --- /dev/null +++ b/exchange/exchangetest/firstpartydata-imp-ext-one-bidder.json @@ -0,0 +1,103 @@ +{ + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "appnexus": { + "placementId": 1 + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "bidder": { + "placementId": 1 + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [{ + "ortbBid": { + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1" + }, + "bidType": "banner" + }] + } + } + } + }, + "response": { + "bids": { + "id": "some-request-id", + "seatbid": [{ + "seat": "appnexus", + "bid": [{ + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1", + "ext": { + "prebid": { + "type": "banner" + } + } + }] + }] + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/firstpartydata-imp-ext-one-prebid-bidder.json b/exchange/exchangetest/firstpartydata-imp-ext-one-prebid-bidder.json new file mode 100644 index 00000000000..1610b9ea47e --- /dev/null +++ b/exchange/exchangetest/firstpartydata-imp-ext-one-prebid-bidder.json @@ -0,0 +1,108 @@ +{ + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "prebid": { + "bidder": { + "appnexus": { + "placementId": 1 + } + } + }, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "some-imp-id", + "banner": { + "format": [{ + "w": 600, + "h": 500 + }, { + "w": 300, + "h": 600 + }] + }, + "ext": { + "bidder": { + "placementId": 1 + }, + "prebid": {}, + "context": { + "data": { + "keywords": "prebid server example" + } + } + } + }] + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "pbsSeatBid": { + "pbsBids": [{ + "ortbBid": { + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1" + }, + "bidType": "banner" + }] + } + } + } + }, + "response": { + "bids": { + "id": "some-request-id", + "seatbid": [{ + "seat": "appnexus", + "bid": [{ + "id": "apn-bid", + "impid": "some-imp-id", + "price": 0.3, + "w": 200, + "h": 500, + "crid": "creative-1", + "ext": { + "prebid": { + "type": "banner" + } + } + }] + }] + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/gdpr-geo-eu-off-device.json b/exchange/exchangetest/gdpr-geo-eu-off-device.json new file mode 100644 index 00000000000..f704cdd5c8e --- /dev/null +++ b/exchange/exchangetest/gdpr-geo-eu-off-device.json @@ -0,0 +1,65 @@ +{ + "assume_gdpr_applies": false, + "gdpr_enabled": true, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id" + }, + "device": { + "geo": { + "country": "FRA" + } + } +} + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "user": { + }, + "device": { + "geo": { + "country": "FRA" + } + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/gdpr-geo-eu-off.json b/exchange/exchangetest/gdpr-geo-eu-off.json new file mode 100644 index 00000000000..24357eb7eec --- /dev/null +++ b/exchange/exchangetest/gdpr-geo-eu-off.json @@ -0,0 +1,61 @@ +{ + "assume_gdpr_applies": false, + "gdpr_enabled": true, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "FRA" + } + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "user": { + "geo": { + "country": "FRA" + } + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/gdpr-geo-eu-on-featureflag-off.json b/exchange/exchangetest/gdpr-geo-eu-on-featureflag-off.json new file mode 100644 index 00000000000..6c6ca3edc62 --- /dev/null +++ b/exchange/exchangetest/gdpr-geo-eu-on-featureflag-off.json @@ -0,0 +1,62 @@ +{ + "assume_gdpr_applies": true, + "gdpr_enabled": false, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "FRA" + } + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "FRA" + } + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/gdpr-geo-eu-on.json b/exchange/exchangetest/gdpr-geo-eu-on.json new file mode 100644 index 00000000000..eb42a17c936 --- /dev/null +++ b/exchange/exchangetest/gdpr-geo-eu-on.json @@ -0,0 +1,61 @@ +{ + "assume_gdpr_applies": true, + "gdpr_enabled": true, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "FRA" + } + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "user": { + "geo": { + "country": "FRA" + } + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/gdpr-geo-usa-off.json b/exchange/exchangetest/gdpr-geo-usa-off.json new file mode 100644 index 00000000000..d56c9318a56 --- /dev/null +++ b/exchange/exchangetest/gdpr-geo-usa-off.json @@ -0,0 +1,61 @@ +{ + "assume_gdpr_applies": false, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "USA" + } + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "USA" + } + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/gdpr-geo-usa-on.json b/exchange/exchangetest/gdpr-geo-usa-on.json new file mode 100644 index 00000000000..f922be9ea4e --- /dev/null +++ b/exchange/exchangetest/gdpr-geo-usa-on.json @@ -0,0 +1,61 @@ +{ + "assume_gdpr_applies": true, + "incomingRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "appnexus": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "USA" + } + } + } + }, + "outgoingRequests": { + "appnexus": { + "expectRequest": { + "ortbRequest": { + "id": "some-request-id", + "site": { + "page": "test.somepage.com" + }, + "imp": [{ + "id": "my-imp-id", + "video": { + "mimes": ["video/mp4"] + }, + "ext": { + "bidder": { + "placementId": 1 + } + } + }], + "user": { + "buyeruid": "some-buyer-id", + "geo": { + "country": "USA" + } + } + }, + "bidAdjustment": 1.0 + }, + "mockResponse": { + "errors": ["appnexus-error"] + } + } + } +} \ No newline at end of file diff --git a/exchange/exchangetest/request-multi-bidders-debug-info.json b/exchange/exchangetest/request-multi-bidders-debug-info.json index db16dbe6013..ec174f75b36 100644 --- a/exchange/exchangetest/request-multi-bidders-debug-info.json +++ b/exchange/exchangetest/request-multi-bidders-debug-info.json @@ -42,7 +42,6 @@ ], "ext": { "prebid": { - "debug":1, "targeting": { "durationRangeSec": [ 15, @@ -204,9 +203,7 @@ }, "test": 1, "ext": { - "prebid": { - "debug": 1, "targeting": { "durationRangeSec": [ 15, diff --git a/exchange/exchangetest/targeting-cache-vast.json b/exchange/exchangetest/targeting-cache-vast.json index f348dd1b29d..53a48c4ec69 100644 --- a/exchange/exchangetest/targeting-cache-vast.json +++ b/exchange/exchangetest/targeting-cache-vast.json @@ -67,7 +67,7 @@ "cache": { "bids": { "cacheId": "0", - "url": "www.pbcserver.com/pbcache/endpoint?uuid=0" + "url": "https://www.pbcserver.com/pbcache/endpoint?uuid=0" }, "key": "", "url": "" diff --git a/exchange/exchangetest/targeting-cache-zero.json b/exchange/exchangetest/targeting-cache-zero.json index 5130153026a..0048ea10917 100644 --- a/exchange/exchangetest/targeting-cache-zero.json +++ b/exchange/exchangetest/targeting-cache-zero.json @@ -70,7 +70,7 @@ "cache": { "bids": { "cacheId": "0", - "url": "www.pbcserver.com/pbcache/endpoint?uuid=0" + "url": "https://www.pbcserver.com/pbcache/endpoint?uuid=0" }, "key": "", "url": "" diff --git a/exchange/impcustomcachekeytest/multiImpVast.json b/exchange/impcustomcachekeytest/multiImpVast.json index bf0b310b04c..db10697431a 100644 --- a/exchange/impcustomcachekeytest/multiImpVast.json +++ b/exchange/impcustomcachekeytest/multiImpVast.json @@ -64,29 +64,29 @@ "bidder": "rubicon" }], "expectedCacheables": [{ - "Type": "xml", - "TTLSeconds": 360, - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://bidoneproducts.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 360, + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://bidoneproducts.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" }, { - "Type": "xml", - "TTLSeconds": 260, - "Key": "34_news_44", - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 260, + "key": "34_news_44", + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://domain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" }, { - "Type": "xml", - "TTLSeconds": 360, - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://anotherdomain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 360, + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://anotherdomain.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" }, { - "Type": "xml", - "TTLSeconds": 3660, - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://bidseveryday.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 3660, + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://bidseveryday.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" }, { - "Type": "xml", - "TTLSeconds": 960, - "Key": "13_sports_22", - "Data":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://anotherbidder.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" + "type": "xml", + "ttlseconds": 960, + "key": "13_sports_22", + "value":"\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://anotherbidder.com/win-notify/1]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e" } ], "defaultTTLs": { diff --git a/exchange/impcustomcachekeytest/multiImpVideo.json b/exchange/impcustomcachekeytest/multiImpVideo.json new file mode 100644 index 00000000000..fc34388875c --- /dev/null +++ b/exchange/impcustomcachekeytest/multiImpVideo.json @@ -0,0 +1,101 @@ +{ + "bidRequest": { + "imp": [{ + "id": "1_0" + }, + { + "id": "1_1" + } + ] + }, + "pbsBids": [{ + "bid": { + "id": "apn_1_0", + "impid": "1_0", + "price": 12.00, + "nurl": "http://apn_1_0.com", + "cat": ["12.00_sports_30s"] + }, + "bidType": "video", + "bidder": "appnexus" + }, { + "bid": { + "id": "spotx_1_0", + "impid": "1_0", + "price": 20.00, + "nurl": "http://spotx_1_0.com", + "cat": ["20_news_30s"] + }, + "bidType": "video", + "bidder": "spotx" + }, { + "bid": { + "id": "apn_1_1", + "impid": "1_1", + "price": 18.00, + "nurl": "http://apn_1_1.com", + "cat": ["18_furniture_30s"] + }, + "bidType": "video", + "bidder": "appnexus" + }, { + "bid": { + "id": "spotx_1_1", + "impid": "1_1", + "price": 17.00, + "nurl": "http://spotx_1_1.com", + "cat": ["17_auto_30s"] + }, + "bidType": "video", + "bidder": "spotx" + }, { + "bid": { + "id": "rubicon_1_1", + "impid": "1_1", + "price": 17.50, + "nurl": "http://rubicon_1_1.com", + "cat": ["17_music_30s"] + }, + "bidType": "video", + "bidder": "rubicon" + }], + "expectedCacheables": [{ + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://apn_1_0.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "12.00_sports_30s" + }, { + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://spotx_1_0.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "20_news_30s" + }, { + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://apn_1_1.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "18_furniture_30s" + }, + { + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://spotx_1_1.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "17_auto_30s" + }, + { + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://rubicon_1_1.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "17_music_30s" + } + ], + "defaultTTLs": { + "banner": 300, + "video": 3600, + "audio": 1800, + "native": 300 + }, + "targetDataIncludeWinners": false, + "targetDataIncludeBidderKeys": true, + "targetDataIncludeCacheBids": false, + "targetDataIncludeCacheVast": true +} diff --git a/exchange/impcustomcachekeytest/multiImpVideoNoIncludeBidderKeys.json b/exchange/impcustomcachekeytest/multiImpVideoNoIncludeBidderKeys.json new file mode 100644 index 00000000000..4dd15344729 --- /dev/null +++ b/exchange/impcustomcachekeytest/multiImpVideoNoIncludeBidderKeys.json @@ -0,0 +1,86 @@ +{ + "bidRequest": { + "imp": [{ + "id": "1_0" + }, + { + "id": "1_1" + } + ] + }, + "pbsBids": [{ + "bid": { + "id": "apn_1_0", + "impid": "1_0", + "price": 12.00, + "nurl": "http://apn_1_0.com", + "cat": ["12.00_sports_30s"] + }, + "bidType": "video", + "bidder": "appnexus" + }, { + "bid": { + "id": "spotx_1_0", + "impid": "1_0", + "price": 20.00, + "nurl": "http://spotx_1_0.com", + "cat": ["20_news_30s"] + }, + "bidType": "video", + "bidder": "spotx" + }, { + "bid": { + "id": "apn_1_1", + "impid": "1_1", + "price": 18.00, + "nurl": "http://apn_1_1.com", + "cat": ["18_furniture_30s"] + }, + "bidType": "video", + "bidder": "appnexus" + }, { + "bid": { + "id": "spotx_1_1", + "impid": "1_1", + "price": 17.00, + "nurl": "http://spotx_1_1.com", + "cat": ["17_auto_30s"] + }, + "bidType": "video", + "bidder": "spotx" + }, { + "bid": { + "id": "rubicon_1_1", + "impid": "1_1", + "price": 17.50, + "nurl": "http://rubicon_1_1.com", + "cat": ["17_music_30s"] + }, + "bidType": "video", + "bidder": "rubicon" + }], + "expectedCacheables": [ + { + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://spotx_1_0.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "20_news_30s" + }, + { + "type": "xml", + "ttlseconds": 3660, + "value": "\u003cVAST version=\"3.0\"\u003e\u003cAd\u003e\u003cWrapper\u003e\u003cAdSystem\u003eprebid.org wrapper\u003c/AdSystem\u003e\u003cVASTAdTagURI\u003e\u003c![CDATA[http://apn_1_1.com]]\u003e\u003c/VASTAdTagURI\u003e\u003cImpression\u003e\u003c/Impression\u003e\u003cCreatives\u003e\u003c/Creatives\u003e\u003c/Wrapper\u003e\u003c/Ad\u003e\u003c/VAST\u003e", + "key": "18_furniture_30s" + } + ], + "defaultTTLs": { + "banner": 300, + "video": 3600, + "audio": 1800, + "native": 300 + }, + "targetDataIncludeWinners": true, + "targetDataIncludeBidderKeys": false, + "targetDataIncludeCacheBids": false, + "targetDataIncludeCacheVast": true +} diff --git a/exchange/legacy.go b/exchange/legacy.go index 619cafbd3b9..43aa3d73b9b 100644 --- a/exchange/legacy.go +++ b/exchange/legacy.go @@ -34,7 +34,7 @@ type adaptedAdapter struct { // // This is not ideal. OpenRTB provides a superset of the legacy data structures. // For requests which use those features, the best we can do is respond with "no bid". -func (bidder *adaptedAdapter) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo, debug bool) (*pbsOrtbSeatBid, []error) { +func (bidder *adaptedAdapter) requestBid(ctx context.Context, request *openrtb.BidRequest, name openrtb_ext.BidderName, bidAdjustment float64, conversions currencies.Conversions, reqInfo *adapters.ExtraRequestInfo) (*pbsOrtbSeatBid, []error) { legacyRequest, legacyBidder, errs := bidder.toLegacyAdapterInputs(request, name) if legacyRequest == nil || legacyBidder == nil { return nil, errs @@ -98,7 +98,7 @@ func (bidder *adaptedAdapter) toLegacyRequest(req *openrtb.BidRequest) (*pbs.PBS } } - if requestExt.Prebid.Debug == 1 { + if requestExt.Prebid.Debug { isDebug = true } diff --git a/exchange/legacy_test.go b/exchange/legacy_test.go index 3c2d1c06ee0..62553ff8a2e 100644 --- a/exchange/legacy_test.go +++ b/exchange/legacy_test.go @@ -4,8 +4,10 @@ import ( "context" "encoding/json" "errors" + "net/http" "reflect" "testing" + "time" "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/adapters" @@ -58,8 +60,8 @@ func TestSiteVideo(t *testing.T) { mockAdapter := mockLegacyAdapter{} exchangeBidder := adaptLegacyAdapter(&mockAdapter) - currencyConverter := currencies.NewRateConverterDefault() - _, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderRubicon, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + _, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderRubicon, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if len(errs) > 0 { t.Errorf("Unexpected error requesting bids: %v", errs) } @@ -92,8 +94,8 @@ func TestAppBanner(t *testing.T) { mockAdapter := mockLegacyAdapter{} exchangeBidder := adaptLegacyAdapter(&mockAdapter) - currencyConverter := currencies.NewRateConverterDefault() - _, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderRubicon, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + _, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderRubicon, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if len(errs) > 0 { t.Errorf("Unexpected error requesting bids: %v", errs) } @@ -138,8 +140,8 @@ func TestBidTransforms(t *testing.T) { } exchangeBidder := adaptLegacyAdapter(&mockAdapter) - currencyConverter := currencies.NewRateConverterDefault() - seatBid, errs := exchangeBidder.requestBid(context.Background(), newAppOrtbRequest(), openrtb_ext.BidderRubicon, bidAdjustment, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + seatBid, errs := exchangeBidder.requestBid(context.Background(), newAppOrtbRequest(), openrtb_ext.BidderRubicon, bidAdjustment, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if len(errs) != 1 { t.Fatalf("Bad error count. Expected 1, got %d", len(errs)) } @@ -287,8 +289,8 @@ func TestErrorResponse(t *testing.T) { } exchangeBidder := adaptLegacyAdapter(&mockAdapter) - currencyConverter := currencies.NewRateConverterDefault() - _, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderRubicon, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + _, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderRubicon, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if len(errs) != 1 { t.Fatalf("Bad error count. Expected 1, got %d", len(errs)) } @@ -326,8 +328,8 @@ func TestWithTargeting(t *testing.T) { }}, } exchangeBidder := adaptLegacyAdapter(&mockAdapter) - currencyConverter := currencies.NewRateConverterDefault() - bid, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderFacebook, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}, false) + currencyConverter := currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)) + bid, errs := exchangeBidder.requestBid(context.Background(), ortbRequest, openrtb_ext.BidderFacebook, 1.0, currencyConverter.Rates(), &adapters.ExtraRequestInfo{}) if len(errs) != 0 { t.Fatalf("This should not produce errors. Got %v", errs) } diff --git a/exchange/price_granularity.go b/exchange/price_granularity.go index ad31f0ae344..ffcce061465 100644 --- a/exchange/price_granularity.go +++ b/exchange/price_granularity.go @@ -7,33 +7,32 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" ) -// DEFAULT_PRECISION should be taken care of in openrtb_ext/request.go, but throwing an additional safety check here. - -// GetCpmStringValue is the externally facing function for computing CPM buckets -func GetCpmStringValue(cpm float64, config openrtb_ext.PriceGranularity) (string, error) { +// GetPriceBucket is the externally facing function for computing CPM buckets +func GetPriceBucket(cpm float64, config openrtb_ext.PriceGranularity) string { cpmStr := "" bucketMax := 0.0 increment := 0.0 precision := config.Precision - // calculate max of highest bucket + for i := 0; i < len(config.Ranges); i++ { if config.Ranges[i].Max > bucketMax { bucketMax = config.Ranges[i].Max } - } // calculate which bucket cpm is in - if cpm > bucketMax { - // If we are over max, just return that - return strconv.FormatFloat(bucketMax, 'f', precision, 64), nil - } - for i := 0; i < len(config.Ranges); i++ { + // find what range cpm is in if cpm >= config.Ranges[i].Min && cpm <= config.Ranges[i].Max { increment = config.Ranges[i].Increment } } - if increment > 0 { + + if cpm > bucketMax { + // We are over max, just return that + cpmStr = strconv.FormatFloat(bucketMax, 'f', precision, 64) + } else if increment > 0 { + // If increment exists, get cpm string value cpmStr = getCpmTarget(cpm, increment, precision) } - return cpmStr, nil + + return cpmStr } func getCpmTarget(cpm float64, increment float64, precision int) string { diff --git a/exchange/price_granularity_test.go b/exchange/price_granularity_test.go index 9c3aa1411d9..6dccc677b7b 100644 --- a/exchange/price_granularity_test.go +++ b/exchange/price_granularity_test.go @@ -1,9 +1,11 @@ package exchange import ( + "math" "testing" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + "github.com/stretchr/testify/assert" ) func TestGetPriceBucketString(t *testing.T) { @@ -28,32 +30,105 @@ func TestGetPriceBucketString(t *testing.T) { }, } - price := 1.87 - getOnePriceBucket(t, "low", low, price, "1.50") - getOnePriceBucket(t, "medium", medium, price, "1.80") - getOnePriceBucket(t, "high", high, price, "1.87") - getOnePriceBucket(t, "auto", auto, price, "1.85") - getOnePriceBucket(t, "dense", dense, price, "1.87") - getOnePriceBucket(t, "custom1", custom1, price, "1.86") - - // test a cpm above the max in low price bucket - price = 5.72 - getOnePriceBucket(t, "low", low, price, "5.00") - getOnePriceBucket(t, "medium", medium, price, "5.70") - getOnePriceBucket(t, "high", high, price, "5.72") - getOnePriceBucket(t, "auto", auto, price, "5.70") - getOnePriceBucket(t, "dense", dense, price, "5.70") - getOnePriceBucket(t, "custom1", custom1, price, "5.70") + // Define test cases + type aTest struct { + granularityId string + granularity openrtb_ext.PriceGranularity + expectedPriceBucket string + } + testGroups := []struct { + groupDesc string + cpm float64 + testCases []aTest + }{ + { + groupDesc: "cpm below the max in every price bucket", + cpm: 1.87, + testCases: []aTest{ + {"low", low, "1.50"}, + {"medium", medium, "1.80"}, + {"high", high, "1.87"}, + {"auto", auto, "1.85"}, + {"dense", dense, "1.87"}, + {"custom1", custom1, "1.86"}, + }, + }, + { + groupDesc: "cpm above the max in low price bucket", + cpm: 5.72, + testCases: []aTest{ + {"low", low, "5.00"}, + {"medium", medium, "5.70"}, + {"high", high, "5.72"}, + {"auto", auto, "5.70"}, + {"dense", dense, "5.70"}, + {"custom1", custom1, "5.70"}, + }, + }, + { + groupDesc: "Precision value corner cases", + cpm: 1.876, + testCases: []aTest{ + { + "Negative precision defaults to number of digits already in CPM float", + openrtb_ext.PriceGranularity{Precision: -1, Ranges: []openrtb_ext.GranularityRange{{Max: 5, Increment: 0.05}}}, + "1.85", + }, + { + "Precision value equals zero, we expect to round up to the nearest integer", + openrtb_ext.PriceGranularity{Ranges: []openrtb_ext.GranularityRange{{Max: 5, Increment: 0.05}}}, + "2", + }, + { + "Largest precision value PBS supports 15", + openrtb_ext.PriceGranularity{Precision: 15, Ranges: []openrtb_ext.GranularityRange{{Max: 5, Increment: 0.05}}}, + "1.850000000000000", + }, + }, + }, + { + groupDesc: "Increment value corner cases", + cpm: 1.876, + testCases: []aTest{ + { + "Negative increment, return empty string", + openrtb_ext.PriceGranularity{Precision: 2, Ranges: []openrtb_ext.GranularityRange{{Max: 5, Increment: -0.05}}}, + "", + }, + { + "Zero increment, return empty string", + openrtb_ext.PriceGranularity{Precision: 2, Ranges: []openrtb_ext.GranularityRange{{Max: 5}}}, + "", + }, + { + "Increment value is greater than CPM itself, return zero float value", + openrtb_ext.PriceGranularity{Precision: 2, Ranges: []openrtb_ext.GranularityRange{{Max: 5, Increment: 1.877}}}, + "0.00", + }, + }, + }, + { + groupDesc: "Negative Cpm, return empty string since it does not belong into any range", + cpm: -1.876, + testCases: []aTest{{"low", low, ""}}, + }, + { + groupDesc: "Zero value Cpm, return the same, only in string format", + cpm: 0, + testCases: []aTest{{"low", low, "0.00"}}, + }, + { + groupDesc: "Large Cpm, return bucket Max", + cpm: math.MaxFloat64, + testCases: []aTest{{"low", low, "5.00"}}, + }, + } -} + for _, testGroup := range testGroups { + for _, test := range testGroup.testCases { + priceBucket := GetPriceBucket(testGroup.cpm, test.granularity) -func getOnePriceBucket(t *testing.T, name string, granularity openrtb_ext.PriceGranularity, price float64, expected string) { - t.Helper() - priceBucket, err := GetCpmStringValue(price, granularity) - if err != nil { - t.Errorf("Granularity: %s :: GetPriceBucketString: %s", name, err.Error()) - } - if priceBucket != expected { - t.Errorf("Granularity: %s :: Expected %s, got %s from %f", name, expected, priceBucket, price) + assert.Equal(t, test.expectedPriceBucket, priceBucket, "Group: %s Granularity: %s :: Expected %s, got %s from %f", testGroup.groupDesc, test.granularityId, test.expectedPriceBucket, priceBucket, testGroup.cpm) + } } } diff --git a/exchange/targeting.go b/exchange/targeting.go index 1bb6a7641ad..31db7114f67 100644 --- a/exchange/targeting.go +++ b/exchange/targeting.go @@ -7,7 +7,7 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" ) -const maxKeyLength = 20 +const MaxKeyLength = 20 // targetData tracks information about the winning Bid in each Imp. // @@ -22,6 +22,8 @@ type targetData struct { includeBidderKeys bool includeCacheBids bool includeCacheVast bool + includeFormat bool + preferDeals bool // cacheHost and cachePath exist to supply cache host and path as targeting parameters cacheHost string cachePath string @@ -53,6 +55,9 @@ func (targData *targetData) setTargeting(auc *auction, isApp bool, categoryMappi if vastID, ok := auc.vastCacheIds[topBidPerBidder.bid]; ok { targData.addKeys(targets, openrtb_ext.HbVastCacheKey, vastID, bidderName, isOverallWinner) } + if targData.includeFormat { + targData.addKeys(targets, openrtb_ext.HbFormatKey, string(topBidPerBidder.bidType), bidderName, isOverallWinner) + } if targData.cacheHost != "" { targData.addKeys(targets, openrtb_ext.HbConstantCacheHostKey, targData.cacheHost, bidderName, isOverallWinner) @@ -79,7 +84,7 @@ func (targData *targetData) setTargeting(auc *auction, isApp bool, categoryMappi func (targData *targetData) addKeys(keys map[string]string, key openrtb_ext.TargetingKey, value string, bidderName openrtb_ext.BidderName, overallWinner bool) { if targData.includeBidderKeys { - keys[key.BidderKey(bidderName, maxKeyLength)] = value + keys[key.BidderKey(bidderName, MaxKeyLength)] = value } if targData.includeWinners && overallWinner { keys[string(key)] = value diff --git a/exchange/targeting_test.go b/exchange/targeting_test.go index 59b92332100..c152f8ea3d6 100644 --- a/exchange/targeting_test.go +++ b/exchange/targeting_test.go @@ -13,7 +13,6 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/gdpr" - "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" metricsConf "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics/config" metricsConfig "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics/config" @@ -50,13 +49,13 @@ func TestTargetingCache(t *testing.T) { // Make sure that the cache keys exist on the bids where they're expected to assertKeyExists(t, bids["winning-bid"], string(openrtb_ext.HbCacheKey), true) - assertKeyExists(t, bids["winning-bid"], openrtb_ext.HbCacheKey.BidderKey(openrtb_ext.BidderAppnexus, maxKeyLength), true) + assertKeyExists(t, bids["winning-bid"], openrtb_ext.HbCacheKey.BidderKey(openrtb_ext.BidderAppnexus, MaxKeyLength), true) assertKeyExists(t, bids["contending-bid"], string(openrtb_ext.HbCacheKey), false) - assertKeyExists(t, bids["contending-bid"], openrtb_ext.HbCacheKey.BidderKey(openrtb_ext.BidderRubicon, maxKeyLength), true) + assertKeyExists(t, bids["contending-bid"], openrtb_ext.HbCacheKey.BidderKey(openrtb_ext.BidderRubicon, MaxKeyLength), true) assertKeyExists(t, bids["losing-bid"], string(openrtb_ext.HbCacheKey), false) - assertKeyExists(t, bids["losing-bid"], openrtb_ext.HbCacheKey.BidderKey(openrtb_ext.BidderAppnexus, maxKeyLength), false) + assertKeyExists(t, bids["losing-bid"], openrtb_ext.HbCacheKey.BidderKey(openrtb_ext.BidderAppnexus, MaxKeyLength), false) //assert hb_cache_host was included assert.Contains(t, string(bids["winning-bid"].Ext), string(openrtb_ext.HbConstantCacheHostKey)) @@ -82,14 +81,20 @@ func runTargetingAuction(t *testing.T, mockBids map[openrtb_ext.BidderName][]*op server := httptest.NewServer(http.HandlerFunc(mockServer)) defer server.Close() + categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") + if error != nil { + t.Errorf("Failed to create a category Fetcher: %v", error) + } + ex := &exchange{ adapterMap: buildAdapterMap(mockBids, server.URL, server.Client()), me: &metricsConf.DummyMetricsEngine{}, cache: &wellBehavedCache{}, cacheTime: time.Duration(0), gDPR: gdpr.AlwaysAllow{}, - currencyConverter: currencies.NewRateConverterDefault(), + currencyConverter: currencies.NewRateConverter(&http.Client{}, "", time.Duration(0)), UsersyncIfAmbiguous: false, + categoriesFetcher: categoriesFetcher, } imps := buildImps(t, mockBids) @@ -104,11 +109,13 @@ func runTargetingAuction(t *testing.T, mockBids map[openrtb_ext.BidderName][]*op req.Site = &openrtb.Site{} } - categoriesFetcher, error := newCategoryFetcher("./test/category-mapping") - if error != nil { - t.Errorf("Failed to create a category Fetcher: %v", error) + auctionRequest := AuctionRequest{ + BidRequest: req, + Account: config.Account{}, + UserSyncs: &emptyUsersync{}, } - bidResp, err := ex.HoldAuction(context.Background(), req, &mockFetcher{}, pbsmetrics.Labels{}, &categoriesFetcher, nil) + + bidResp, err := ex.HoldAuction(context.Background(), auctionRequest, nil) if err != nil { t.Fatalf("Unexpected errors running auction: %v", err) @@ -134,7 +141,7 @@ func buildAdapterMap(bids map[openrtb_ext.BidderName][]*openrtb.Bid, mockServerU adapterMap[bidder] = adaptBidder(&mockTargetingBidder{ mockServerURL: mockServerURL, bids: bids, - }, client, &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}) + }, client, &config.Configuration{}, &metricsConfig.DummyMetricsEngine{}, openrtb_ext.BidderAppnexus) } return adapterMap } @@ -238,12 +245,215 @@ func (m *mockTargetingBidder) MakeBids(internalRequest *openrtb.BidRequest, exte return bidResponse, nil } -type mockFetcher struct{} +func mockServer(w http.ResponseWriter, req *http.Request) { + w.Write([]byte("{}")) +} -func (f *mockFetcher) GetId(bidder openrtb_ext.BidderName) (string, bool) { - return "", false +type TargetingTestData struct { + Description string + TargetData targetData + Auction auction + IsApp bool + CategoryMapping map[string]string + ExpectedBidTargetsByBidder map[string]map[openrtb_ext.BidderName]map[string]string } -func mockServer(w http.ResponseWriter, req *http.Request) { - w.Write([]byte("{}")) +var bid123 *openrtb.Bid = &openrtb.Bid{ + Price: 1.23, +} + +var bid111 *openrtb.Bid = &openrtb.Bid{ + Price: 1.11, + DealID: "mydeal", +} +var bid084 *openrtb.Bid = &openrtb.Bid{ + Price: 0.84, +} + +var TargetingTests []TargetingTestData = []TargetingTestData{ + { + Description: "Targeting winners only (most basic targeting example)", + TargetData: targetData{ + priceGranularity: openrtb_ext.PriceGranularityFromString("med"), + includeWinners: true, + }, + Auction: auction{ + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + bid: bid123, + bidType: openrtb_ext.BidTypeBanner, + }, + openrtb_ext.BidderRubicon: { + bid: bid084, + bidType: openrtb_ext.BidTypeBanner, + }, + }, + }, + }, + ExpectedBidTargetsByBidder: map[string]map[openrtb_ext.BidderName]map[string]string{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + "hb_bidder": "appnexus", + "hb_pb": "1.20", + }, + openrtb_ext.BidderRubicon: {}, + }, + }, + }, + { + Description: "Targeting on bidders only", + TargetData: targetData{ + priceGranularity: openrtb_ext.PriceGranularityFromString("med"), + includeBidderKeys: true, + }, + Auction: auction{ + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + bid: bid123, + bidType: openrtb_ext.BidTypeBanner, + }, + openrtb_ext.BidderRubicon: { + bid: bid084, + bidType: openrtb_ext.BidTypeBanner, + }, + }, + }, + }, + ExpectedBidTargetsByBidder: map[string]map[openrtb_ext.BidderName]map[string]string{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + "hb_bidder_appnexus": "appnexus", + "hb_pb_appnexus": "1.20", + }, + openrtb_ext.BidderRubicon: { + "hb_bidder_rubicon": "rubicon", + "hb_pb_rubicon": "0.80", + }, + }, + }, + }, + { + Description: "Full basic targeting with hd_format", + TargetData: targetData{ + priceGranularity: openrtb_ext.PriceGranularityFromString("med"), + includeWinners: true, + includeBidderKeys: true, + includeFormat: true, + }, + Auction: auction{ + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + bid: bid123, + bidType: openrtb_ext.BidTypeBanner, + }, + openrtb_ext.BidderRubicon: { + bid: bid084, + bidType: openrtb_ext.BidTypeBanner, + }, + }, + }, + }, + ExpectedBidTargetsByBidder: map[string]map[openrtb_ext.BidderName]map[string]string{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + "hb_bidder": "appnexus", + "hb_bidder_appnexus": "appnexus", + "hb_pb": "1.20", + "hb_pb_appnexus": "1.20", + "hb_format": "banner", + "hb_format_appnexus": "banner", + }, + openrtb_ext.BidderRubicon: { + "hb_bidder_rubicon": "rubicon", + "hb_pb_rubicon": "0.80", + "hb_format_rubicon": "banner", + }, + }, + }, + }, + { + Description: "Cache and deal targeting test", + TargetData: targetData{ + priceGranularity: openrtb_ext.PriceGranularityFromString("med"), + includeBidderKeys: true, + cacheHost: "cache.prebid.com", + cachePath: "cache", + }, + Auction: auction{ + winningBidsByBidder: map[string]map[openrtb_ext.BidderName]*pbsOrtbBid{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + bid: bid123, + bidType: openrtb_ext.BidTypeBanner, + }, + openrtb_ext.BidderRubicon: { + bid: bid111, + bidType: openrtb_ext.BidTypeBanner, + }, + }, + }, + cacheIds: map[*openrtb.Bid]string{ + bid123: "55555", + bid111: "cacheme", + }, + }, + ExpectedBidTargetsByBidder: map[string]map[openrtb_ext.BidderName]map[string]string{ + "ImpId-1": { + openrtb_ext.BidderAppnexus: { + "hb_bidder_appnexus": "appnexus", + "hb_pb_appnexus": "1.20", + "hb_cache_id_appnexus": "55555", + "hb_cache_host_appnex": "cache.prebid.com", + "hb_cache_path_appnex": "cache", + }, + openrtb_ext.BidderRubicon: { + "hb_bidder_rubicon": "rubicon", + "hb_pb_rubicon": "1.10", + "hb_cache_id_rubicon": "cacheme", + "hb_deal_rubicon": "mydeal", + "hb_cache_host_rubico": "cache.prebid.com", + "hb_cache_path_rubico": "cache", + }, + }, + }, + }, +} + +func TestSetTargeting(t *testing.T) { + for _, test := range TargetingTests { + auc := &test.Auction + // Set rounded prices from the auction data + auc.setRoundedPrices(test.TargetData.priceGranularity) + winningBids := make(map[string]*pbsOrtbBid) + // Set winning bids from the auction data + for imp, bidsByBidder := range auc.winningBidsByBidder { + for _, bid := range bidsByBidder { + if winningBid, ok := winningBids[imp]; ok { + if winningBid.bid.Price < bid.bid.Price { + winningBids[imp] = bid + } + } else { + winningBids[imp] = bid + } + } + } + auc.winningBids = winningBids + targData := test.TargetData + targData.setTargeting(auc, test.IsApp, test.CategoryMapping) + for imp, targetsByBidder := range test.ExpectedBidTargetsByBidder { + for bidder, expected := range targetsByBidder { + assert.Equal(t, + expected, + auc.winningBidsByBidder[imp][bidder].bidTargets, + "Test: %s\nTargeting failed for bidder %s on imp %s.", + test.Description, + string(bidder), + imp) + } + } + } + } diff --git a/exchange/utils.go b/exchange/utils.go index 969471b04c3..78baa00ad9f 100644 --- a/exchange/utils.go +++ b/exchange/utils.go @@ -6,6 +6,8 @@ import ( "fmt" "math/rand" + "github.com/prebid/go-gdpr/vendorconsent" + "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/gdpr" @@ -17,6 +19,34 @@ import ( "github.com/buger/jsonparser" ) +var integrationTypeMap = map[pbsmetrics.RequestType]config.IntegrationType{ + pbsmetrics.ReqTypeAMP: config.IntegrationTypeAMP, + pbsmetrics.ReqTypeORTB2App: config.IntegrationTypeApp, + pbsmetrics.ReqTypeVideo: config.IntegrationTypeVideo, + pbsmetrics.ReqTypeORTB2Web: config.IntegrationTypeWeb, +} + +const unknownBidder string = "" + +func BidderToPrebidSChains(req *openrtb_ext.ExtRequest) (map[string]*openrtb_ext.ExtRequestPrebidSChainSChain, error) { + bidderToSChains := make(map[string]*openrtb_ext.ExtRequestPrebidSChainSChain) + + if req != nil { + for _, schainWrapper := range req.Prebid.SChains { + for _, bidder := range schainWrapper.Bidders { + if _, present := bidderToSChains[bidder]; present { + return nil, fmt.Errorf("request.ext.prebid.schains contains multiple schains for bidder %s; "+ + "it must contain no more than one per bidder.", bidder) + } else { + bidderToSChains[bidder] = &schainWrapper.SChain + } + } + } + } + + return bidderToSChains, nil +} + // cleanOpenRTBRequests splits the input request into requests which are sanitized for each bidder. Intended behavior is: // // 1. BidRequest.Imp[].Ext will only contain the "prebid" field and a "bidder" field which has the params for the intended Bidder. @@ -24,12 +54,14 @@ import ( // 3. BidRequest.User.BuyerUID will be set to that Bidder's ID. func cleanOpenRTBRequests(ctx context.Context, orig *openrtb.BidRequest, + requestExt *openrtb_ext.ExtRequest, usersyncs IdFetcher, blables map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels, labels pbsmetrics.Labels, gDPR gdpr.Permissions, usersyncIfAmbiguous bool, - privacyConfig config.Privacy) (requestsByBidder map[openrtb_ext.BidderName]*openrtb.BidRequest, aliases map[string]string, errs []error) { + privacyConfig config.Privacy, + account *config.Account) (requestsByBidder map[openrtb_ext.BidderName]*openrtb.BidRequest, aliases map[string]string, privacyLabels pbsmetrics.PrivacyLabels, errs []error) { impsByBidder, errs := splitImps(orig.Imp) if len(errs) > 0 { @@ -41,42 +73,62 @@ func cleanOpenRTBRequests(ctx context.Context, return } - requestsByBidder, errs = splitBidRequest(orig, impsByBidder, aliases, usersyncs, blables, labels) + requestsByBidder, errs = splitBidRequest(orig, requestExt, impsByBidder, aliases, usersyncs, blables, labels) + + if len(requestsByBidder) == 0 { + return + } gdpr := extractGDPR(orig, usersyncIfAmbiguous) consent := extractConsent(orig) ampGDPRException := (labels.RType == pbsmetrics.ReqTypeAMP) && gDPR.AMPException() - var ccpaPolicy ccpa.Policy - if privacyConfig.CCPA.Enforce { - ccpaPolicy, _ = ccpa.ReadPolicy(orig) + ccpaEnforcer, err := extractCCPA(orig, privacyConfig, account, aliases, integrationTypeMap[labels.RType]) + if err != nil { + errs = append(errs, err) + return } - var lmtPolicy lmt.Policy - if privacyConfig.LMT.Enforce { - lmtPolicy = lmt.ReadPolicy(orig) - } + lmtEnforcer := extractLMT(orig, privacyConfig) // request level privacy policies privacyEnforcement := privacy.Enforcement{ - CCPA: ccpaPolicy.ShouldEnforce(), COPPA: orig.Regs != nil && orig.Regs.COPPA == 1, - LMT: lmtPolicy.ShouldEnforce(), + LMT: lmtEnforcer.ShouldEnforce(unknownBidder), + } + + privacyLabels.CCPAProvided = ccpaEnforcer.CanEnforce() + privacyLabels.CCPAEnforced = ccpaEnforcer.ShouldEnforce(unknownBidder) + privacyLabels.COPPAEnforced = privacyEnforcement.COPPA + privacyLabels.LMTEnforced = lmtEnforcer.ShouldEnforce(unknownBidder) + + gdprEnabled := gdprEnabled(account, privacyConfig, integrationTypeMap[labels.RType]) + + if gdpr == 1 && gdprEnabled { + privacyLabels.GDPREnforced = true + parsedConsent, err := vendorconsent.ParseString(consent) + if err == nil { + version := int(parsedConsent.Version()) + privacyLabels.GDPRTCFVersion = pbsmetrics.TCFVersionToValue(version) + } } // bidder level privacy policies for bidder, bidReq := range requestsByBidder { + // CCPA + privacyEnforcement.CCPA = ccpaEnforcer.ShouldEnforce(bidder.String()) - if gdpr == 1 { + // GDPR + if gdpr == 1 && gdprEnabled { coreBidder := resolveBidder(bidder.String(), aliases) var publisherID = labels.PubID - ok, geo, err := gDPR.PersonalInfoAllowed(ctx, coreBidder, publisherID, consent) - privacyEnforcement.GDPR = !ok && err == nil + _, geo, id, err := gDPR.PersonalInfoAllowed(ctx, coreBidder, publisherID, consent) privacyEnforcement.GDPRGeo = !geo && err == nil + privacyEnforcement.GDPRID = !id && err == nil } else { - privacyEnforcement.GDPR = false privacyEnforcement.GDPRGeo = false + privacyEnforcement.GDPRID = false } privacyEnforcement.Apply(bidReq, ampGDPRException) @@ -85,6 +137,46 @@ func cleanOpenRTBRequests(ctx context.Context, return } +func gdprEnabled(account *config.Account, privacyConfig config.Privacy, integrationType config.IntegrationType) bool { + if accountEnabled := account.GDPR.EnabledForIntegrationType(integrationType); accountEnabled != nil { + return *accountEnabled + } + return privacyConfig.GDPR.Enabled +} + +func ccpaEnabled(account *config.Account, privacyConfig config.Privacy, requestType config.IntegrationType) bool { + if accountEnabled := account.CCPA.EnabledForIntegrationType(requestType); accountEnabled != nil { + return *accountEnabled + } + return privacyConfig.CCPA.Enforce +} + +func extractCCPA(orig *openrtb.BidRequest, privacyConfig config.Privacy, account *config.Account, aliases map[string]string, requestType config.IntegrationType) (privacy.PolicyEnforcer, error) { + ccpaPolicy, err := ccpa.ReadFromRequest(orig) + if err != nil { + return privacy.NilPolicyEnforcer{}, err + } + + validBidders := GetValidBidders(aliases) + ccpaParsedPolicy, err := ccpaPolicy.Parse(validBidders) + if err != nil { + return privacy.NilPolicyEnforcer{}, err + } + + ccpaEnforcer := privacy.EnabledPolicyEnforcer{ + Enabled: ccpaEnabled(account, privacyConfig, requestType), + PolicyEnforcer: ccpaParsedPolicy, + } + return ccpaEnforcer, nil +} + +func extractLMT(orig *openrtb.BidRequest, privacyConfig config.Privacy) privacy.PolicyEnforcer { + return privacy.EnabledPolicyEnforcer{ + Enabled: privacyConfig.LMT.Enforce, + PolicyEnforcer: lmt.ReadFromRequest(orig), + } +} + func getBidderExts(reqExt *openrtb_ext.ExtRequest) (map[string]map[string]interface{}, error) { if reqExt == nil { return nil, nil @@ -108,7 +200,14 @@ func getBidderExts(reqExt *openrtb_ext.ExtRequest) (map[string]map[string]interf return bidderParams, nil } -func splitBidRequest(req *openrtb.BidRequest, impsByBidder map[string][]openrtb.Imp, aliases map[string]string, usersyncs IdFetcher, blabels map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels, labels pbsmetrics.Labels) (map[openrtb_ext.BidderName]*openrtb.BidRequest, []error) { +func splitBidRequest(req *openrtb.BidRequest, + requestExt *openrtb_ext.ExtRequest, + impsByBidder map[string][]openrtb.Imp, + aliases map[string]string, + usersyncs IdFetcher, + blabels map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels, + labels pbsmetrics.Labels) (map[openrtb_ext.BidderName]*openrtb.BidRequest, []error) { + requestsByBidder := make(map[openrtb_ext.BidderName]*openrtb.BidRequest, len(impsByBidder)) explicitBuyerUIDs, err := extractBuyerUIDs(req.User) if err != nil { @@ -116,17 +215,23 @@ func splitBidRequest(req *openrtb.BidRequest, impsByBidder map[string][]openrtb. } var bidderExt map[string]map[string]interface{} - var reqExt openrtb_ext.ExtRequest - if req.Ext != nil { - err = json.Unmarshal(req.Ext, &reqExt) + if requestExt != nil { + bidderExt, err = getBidderExts(requestExt) if err != nil { return nil, []error{err} } + } - bidderExt, err = getBidderExts(&reqExt) - if err != nil { - return nil, []error{err} - } + var sChainsByBidder map[string]*openrtb_ext.ExtRequestPrebidSChainSChain + + sChainsByBidder, err = BidderToPrebidSChains(requestExt) + if err != nil { + return nil, []error{err} + } + + reqExt, err := getExtJson(req, requestExt) + if err != nil { + return nil, []error{err} } for bidder, imps := range impsByBidder { @@ -147,18 +252,23 @@ func splitBidRequest(req *openrtb.BidRequest, impsByBidder map[string][]openrtb. } else { blabels[coreBidder].CookieFlag = pbsmetrics.CookieFlagYes } + reqCopy.Imp = imps + prepareSource(&reqCopy, bidder, sChainsByBidder) + if len(bidderExt) != 0 { bidderName := openrtb_ext.BidderName(bidder) if bidderParams, ok := bidderExt[string(bidderName)]; ok { - reqExt.Prebid.BidderParams = bidderParams + requestExt.Prebid.BidderParams = bidderParams } else { - reqExt.Prebid.BidderParams = nil + requestExt.Prebid.BidderParams = nil } - if reqCopy.Ext, err = json.Marshal(&reqExt); err != nil { + if reqCopy.Ext, err = getExtJson(req, requestExt); err != nil { return nil, []error{err} } + } else { + reqCopy.Ext = reqExt } requestsByBidder[openrtb_ext.BidderName(bidder)] = &reqCopy @@ -166,6 +276,47 @@ func splitBidRequest(req *openrtb.BidRequest, impsByBidder map[string][]openrtb. return requestsByBidder, nil } +func getExtJson(req *openrtb.BidRequest, unpackedExt *openrtb_ext.ExtRequest) (json.RawMessage, error) { + if len(req.Ext) == 0 || unpackedExt == nil { + return json.RawMessage(``), nil + } + + extCopy := *unpackedExt + extCopy.Prebid.SChains = nil + return json.Marshal(extCopy) +} + +func prepareSource(req *openrtb.BidRequest, bidder string, sChainsByBidder map[string]*openrtb_ext.ExtRequestPrebidSChainSChain) { + const sChainWildCard = "*" + var selectedSChain *openrtb_ext.ExtRequestPrebidSChainSChain + + wildCardSChain := sChainsByBidder[sChainWildCard] + bidderSChain := sChainsByBidder[bidder] + + // source should not be modified + if bidderSChain == nil && wildCardSChain == nil { + return + } + + if bidderSChain != nil { + selectedSChain = bidderSChain + } else { + selectedSChain = wildCardSChain + } + + // set source + if req.Source == nil { + req.Source = &openrtb.Source{} + } + schain := openrtb_ext.ExtRequestPrebidSChain{ + SChain: *selectedSChain, + } + sourceExt, err := json.Marshal(schain) + if err == nil { + req.Source.Ext = sourceExt + } +} + // extractBuyerUIDs parses the values from user.ext.prebid.buyeruids, and then deletes those values from the ext. // This prevents a Bidder from using these values to figure out who else is involved in the Auction. func extractBuyerUIDs(user *openrtb.User) (map[string]string, error) { @@ -221,13 +372,18 @@ func splitImps(imps []openrtb.Imp) (map[string][]openrtb.Imp, []error) { imp := imps[i] impExt := impExts[i] + var firstPartyDataContext json.RawMessage + if context, exists := impExt[openrtb_ext.FirstPartyDataContextExtKey]; exists { + firstPartyDataContext = context + } + rawPrebidExt, ok := impExt[openrtb_ext.PrebidExtKey] if ok { var prebidExt openrtb_ext.ExtImpPrebid if err := json.Unmarshal(rawPrebidExt, &prebidExt); err == nil && prebidExt.Bidder != nil { - if errs := sanitizedImpCopy(&imp, prebidExt.Bidder, rawPrebidExt, &splitImps); errs != nil { + if errs := sanitizedImpCopy(&imp, prebidExt.Bidder, rawPrebidExt, firstPartyDataContext, &splitImps); errs != nil { errList = append(errList, errs...) } @@ -235,7 +391,7 @@ func splitImps(imps []openrtb.Imp) (map[string][]openrtb.Imp, []error) { } } - if errs := sanitizedImpCopy(&imp, impExt, rawPrebidExt, &splitImps); errs != nil { + if errs := sanitizedImpCopy(&imp, impExt, rawPrebidExt, firstPartyDataContext, &splitImps); errs != nil { errList = append(errList, errs...) } } @@ -243,35 +399,38 @@ func splitImps(imps []openrtb.Imp) (map[string][]openrtb.Imp, []error) { return splitImps, nil } -// sanitizedImpCopy returns a copy of imp with its ext filtered so that only "prebid" and bidder params exist. +// sanitizedImpCopy returns a copy of imp with its ext filtered so that only "prebid", "context", and bidder params exist. // It will not mutate the input imp. // This function will write the new imps to the output map passed in func sanitizedImpCopy(imp *openrtb.Imp, bidderExts map[string]json.RawMessage, rawPrebidExt json.RawMessage, + firstPartyDataContext json.RawMessage, out *map[string][]openrtb.Imp) []error { var prebidExt map[string]json.RawMessage var errs []error - // We don't want to include other demand partners' bidder params - // in the sanitized imp if err := json.Unmarshal(rawPrebidExt, &prebidExt); err == nil { - delete(prebidExt, "bidder") - - var err error - if rawPrebidExt, err = json.Marshal(prebidExt); err != nil { - errs = append(errs, err) + // Remove the entire bidder field. We will already have the content we need in bidderExts. We + // don't want to include other demand partners' bidder params in the sanitized imp. + if _, hasBidderField := prebidExt["bidder"]; hasBidderField { + delete(prebidExt, "bidder") + + var err error + if rawPrebidExt, err = json.Marshal(prebidExt); err != nil { + errs = append(errs, err) + } } } for bidder, ext := range bidderExts { - if bidder == openrtb_ext.PrebidExtKey { + if bidder == openrtb_ext.PrebidExtKey || bidder == openrtb_ext.FirstPartyDataContextExtKey { continue } impCopy := *imp - newExt := make(map[string]json.RawMessage, 2) + newExt := make(map[string]json.RawMessage, 3) newExt["bidder"] = ext @@ -279,6 +438,10 @@ func sanitizedImpCopy(imp *openrtb.Imp, newExt[openrtb_ext.PrebidExtKey] = rawPrebidExt } + if len(firstPartyDataContext) > 0 { + newExt[openrtb_ext.FirstPartyDataContextExtKey] = firstPartyDataContext + } + rawExt, err := json.Marshal(newExt) if err != nil { errs = append(errs, err) @@ -340,7 +503,7 @@ func resolveBidder(bidder string, aliases map[string]string) openrtb_ext.BidderN } // parseImpExts does a partial-unmarshal of the imp[].Ext field. -// The keys in the returned map are expected to be "prebid", core BidderNames, or Aliases for this request. +// The keys in the returned map are expected to be "prebid", "context", core BidderNames, or Aliases for this request. func parseImpExts(imps []openrtb.Imp) ([]map[string]json.RawMessage, error) { exts := make([]map[string]json.RawMessage, len(imps)) // Loop over every impression in the request @@ -367,6 +530,20 @@ func parseAliases(orig *openrtb.BidRequest) (map[string]string, []error) { return aliases, nil } +func GetValidBidders(aliases map[string]string) map[string]struct{} { + validBidders := make(map[string]struct{}) + + for _, v := range openrtb_ext.BidderMap { + validBidders[v.String()] = struct{}{} + } + + for k := range aliases { + validBidders[k] = struct{}{} + } + + return validBidders +} + // Quick little randomizer for a list of strings. Stuffing it in utils to keep other files clean func randomizeList(list []openrtb_ext.BidderName) { l := len(list) @@ -377,3 +554,78 @@ func randomizeList(list []openrtb_ext.BidderName) { list[i], list[j] = list[j], list[i] } } + +func extractBidRequestExt(bidRequest *openrtb.BidRequest) (*openrtb_ext.ExtRequest, error) { + requestExt := &openrtb_ext.ExtRequest{} + + if bidRequest == nil { + return requestExt, fmt.Errorf("Error bidRequest should not be nil") + } + + if len(bidRequest.Ext) > 0 { + err := json.Unmarshal(bidRequest.Ext, &requestExt) + if err != nil { + return requestExt, fmt.Errorf("Error decoding Request.ext : %s", err.Error()) + } + } + return requestExt, nil +} + +func getExtCacheInstructions(requestExt *openrtb_ext.ExtRequest) extCacheInstructions { + //returnCreative defaults to true + cacheInstructions := extCacheInstructions{returnCreative: true} + foundBidsRC := false + foundVastRC := false + + if requestExt != nil && requestExt.Prebid.Cache != nil { + if requestExt.Prebid.Cache.Bids != nil { + cacheInstructions.cacheBids = true + if requestExt.Prebid.Cache.Bids.ReturnCreative != nil { + cacheInstructions.returnCreative = *requestExt.Prebid.Cache.Bids.ReturnCreative + foundBidsRC = true + } + } + if requestExt.Prebid.Cache.VastXML != nil { + cacheInstructions.cacheVAST = true + if requestExt.Prebid.Cache.VastXML.ReturnCreative != nil { + cacheInstructions.returnCreative = *requestExt.Prebid.Cache.VastXML.ReturnCreative + foundVastRC = true + } + } + } + + if foundBidsRC && foundVastRC { + cacheInstructions.returnCreative = *requestExt.Prebid.Cache.Bids.ReturnCreative || *requestExt.Prebid.Cache.VastXML.ReturnCreative + } + + return cacheInstructions +} + +func getExtTargetData(requestExt *openrtb_ext.ExtRequest, cacheInstructions *extCacheInstructions) *targetData { + var targData *targetData + + if requestExt != nil && requestExt.Prebid.Targeting != nil { + targData = &targetData{ + priceGranularity: requestExt.Prebid.Targeting.PriceGranularity, + includeWinners: requestExt.Prebid.Targeting.IncludeWinners, + includeBidderKeys: requestExt.Prebid.Targeting.IncludeBidderKeys, + includeCacheBids: cacheInstructions.cacheBids, + includeCacheVast: cacheInstructions.cacheVAST, + includeFormat: requestExt.Prebid.Targeting.IncludeFormat, + preferDeals: requestExt.Prebid.Targeting.PreferDeals, + } + } + return targData +} + +func getDebugInfo(bidRequest *openrtb.BidRequest, requestExt *openrtb_ext.ExtRequest) bool { + return (bidRequest != nil && bidRequest.Test == 1) || (requestExt != nil && requestExt.Prebid.Debug) +} + +func getExtBidAdjustmentFactors(requestExt *openrtb_ext.ExtRequest) map[string]float64 { + var bidAdjustmentFactors map[string]float64 + if requestExt != nil { + bidAdjustmentFactors = requestExt.Prebid.BidAdjustmentFactors + } + return bidAdjustmentFactors +} diff --git a/exchange/utils_test.go b/exchange/utils_test.go index bd1be73ff3b..202b479fecd 100644 --- a/exchange/utils_test.go +++ b/exchange/utils_test.go @@ -3,10 +3,13 @@ package exchange import ( "context" "encoding/json" + "errors" + "fmt" "testing" "github.com/PubMatic-OpenWrap/openrtb" "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" "github.com/stretchr/testify/assert" @@ -15,7 +18,9 @@ import ( // permissionsMock mocks the Permissions interface for tests // // It only allows appnexus for GDPR consent -type permissionsMock struct{} +type permissionsMock struct { + personalInfoAllowed bool +} func (p *permissionsMock) HostCookiesAllowed(ctx context.Context, consent string) (bool, error) { return true, nil @@ -25,11 +30,8 @@ func (p *permissionsMock) BidderSyncAllowed(ctx context.Context, bidder openrtb_ return true, nil } -func (p *permissionsMock) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) { - if bidder == "appnexus" { - return true, true, nil - } - return false, false, nil +func (p *permissionsMock) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { + return p.personalInfoAllowed, p.personalInfoAllowed, p.personalInfoAllowed, nil } func (p *permissionsMock) AMPException() bool { @@ -80,7 +82,7 @@ func TestCleanOpenRTBRequests(t *testing.T) { } for _, test := range testCases { - reqByBidders, _, err := cleanOpenRTBRequests(context.Background(), test.req, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{}, true, privacyConfig) + reqByBidders, _, _, err := cleanOpenRTBRequests(context.Background(), test.req, nil, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{personalInfoAllowed: true}, true, privacyConfig, &config.Account{}) if test.hasError { assert.NotNil(t, err, "Error shouldn't be nil") } else { @@ -91,40 +93,154 @@ func TestCleanOpenRTBRequests(t *testing.T) { } func TestCleanOpenRTBRequestsCCPA(t *testing.T) { + trueValue, falseValue := true, false + testCases := []struct { - description string - enforceCCPA bool - expectDataScrub bool + description string + reqExt json.RawMessage + ccpaConsent string + ccpaHostEnabled bool + ccpaAccountEnabled *bool + expectDataScrub bool + expectPrivacyLabels pbsmetrics.PrivacyLabels }{ { - description: "Feature Flag Enabled", - enforceCCPA: true, - expectDataScrub: true, + description: "Feature Flags Enabled - Opt Out", + ccpaConsent: "1-Y-", + ccpaHostEnabled: true, + ccpaAccountEnabled: &trueValue, + expectDataScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: true, + }, }, { - description: "Feature Flag Disabled", - enforceCCPA: false, - expectDataScrub: false, + description: "Feature Flags Enabled - Opt In", + ccpaConsent: "1-N-", + ccpaHostEnabled: true, + ccpaAccountEnabled: &trueValue, + expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: false, + }, + }, + { + description: "Feature Flags Enabled - No Sale Star - Doesn't Scrub", + reqExt: json.RawMessage(`{"prebid":{"nosale":["*"]}}`), + ccpaConsent: "1-Y-", + ccpaHostEnabled: true, + ccpaAccountEnabled: &trueValue, + expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: false, + }, + }, + { + description: "Feature Flags Enabled - No Sale Specific Bidder - Doesn't Scrub", + reqExt: json.RawMessage(`{"prebid":{"nosale":["appnexus"]}}`), + ccpaConsent: "1-Y-", + ccpaHostEnabled: true, + ccpaAccountEnabled: &trueValue, + expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: true, + }, + }, + { + description: "Feature Flags Enabled - No Sale Different Bidder - Scrubs", + reqExt: json.RawMessage(`{"prebid":{"nosale":["rubicon"]}}`), + ccpaConsent: "1-Y-", + ccpaHostEnabled: true, + ccpaAccountEnabled: &trueValue, + expectDataScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: true, + }, + }, + { + description: "Feature flags Account CCPA enabled, host CCPA disregarded - Opt Out", + ccpaConsent: "1-Y-", + ccpaHostEnabled: false, + ccpaAccountEnabled: &trueValue, + expectDataScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: true, + }, + }, + { + description: "Feature flags Account CCPA disabled, host CCPA disregarded", + ccpaConsent: "1-Y-", + ccpaHostEnabled: true, + ccpaAccountEnabled: &falseValue, + expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: false, + }, + }, + { + description: "Feature flags Account CCPA not specified, host CCPA enabled - Opt Out", + ccpaConsent: "1-Y-", + ccpaHostEnabled: true, + ccpaAccountEnabled: nil, + expectDataScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: true, + }, + }, + { + description: "Feature flags Account CCPA not specified, host CCPA disabled", + ccpaConsent: "1-Y-", + ccpaHostEnabled: false, + ccpaAccountEnabled: nil, + expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + CCPAProvided: true, + CCPAEnforced: false, + }, }, } for _, test := range testCases { req := newBidRequest(t) + req.Ext = test.reqExt req.Regs = &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"1-Y-"}`), + Ext: json.RawMessage(`{"us_privacy":"` + test.ccpaConsent + `"}`), } privacyConfig := config.Privacy{ CCPA: config.CCPA{ - Enforce: test.enforceCCPA, + Enforce: test.ccpaHostEnabled, }, } - results, _, errs := cleanOpenRTBRequests(context.Background(), req, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{}, true, privacyConfig) + accountConfig := config.Account{ + CCPA: config.AccountCCPA{ + Enabled: test.ccpaAccountEnabled, + }, + } + + results, _, privacyLabels, errs := cleanOpenRTBRequests( + context.Background(), + req, + nil, + &emptyUsersync{}, + map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, + pbsmetrics.Labels{}, + &permissionsMock{personalInfoAllowed: true}, + true, + privacyConfig, + &accountConfig) result := results["appnexus"] assert.Nil(t, errs) - if test.expectDataScrub { assert.Equal(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") assert.Equal(t, result.Device.DIDMD5, "", test.description+":Device.DIDMD5") @@ -132,6 +248,696 @@ func TestCleanOpenRTBRequestsCCPA(t *testing.T) { assert.NotEqual(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") assert.NotEqual(t, result.Device.DIDMD5, "", test.description+":Device.DIDMD5") } + assert.Equal(t, test.expectPrivacyLabels, privacyLabels, test.description+":PrivacyLabels") + } +} + +func TestCleanOpenRTBRequestsCCPAErrors(t *testing.T) { + testCases := []struct { + description string + reqExt json.RawMessage + reqRegsExt json.RawMessage + expectError error + }{ + { + description: "Invalid Consent", + reqExt: json.RawMessage(`{"prebid":{"nosale":["*"]}}`), + reqRegsExt: json.RawMessage(`{"us_privacy":"malformed"}`), + expectError: &errortypes.InvalidPrivacyConsent{"request.regs.ext.us_privacy must contain 4 characters"}, + }, + { + description: "Invalid No Sale Bidders", + reqExt: json.RawMessage(`{"prebid":{"nosale":["*", "another"]}}`), + reqRegsExt: json.RawMessage(`{"us_privacy":"1NYN"}`), + expectError: errors.New("request.ext.prebid.nosale is invalid: can only specify all bidders if no other bidders are provided"), + }, + } + + for _, test := range testCases { + req := newBidRequest(t) + req.Ext = test.reqExt + req.Regs = &openrtb.Regs{Ext: test.reqRegsExt} + + var reqExtStruct openrtb_ext.ExtRequest + err := json.Unmarshal(req.Ext, &reqExtStruct) + assert.NoError(t, err, test.description+":marshal_ext") + + privacyConfig := config.Privacy{ + CCPA: config.CCPA{ + Enforce: true, + }, + } + _, _, _, errs := cleanOpenRTBRequests(context.Background(), req, &reqExtStruct, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{personalInfoAllowed: true}, true, privacyConfig, &config.Account{}) + + assert.ElementsMatch(t, []error{test.expectError}, errs, test.description) + } +} + +func TestCleanOpenRTBRequestsCOPPA(t *testing.T) { + testCases := []struct { + description string + coppa int8 + expectDataScrub bool + expectPrivacyLabels pbsmetrics.PrivacyLabels + }{ + { + description: "Enabled", + coppa: 1, + expectDataScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + COPPAEnforced: true, + }, + }, + { + description: "Disabled", + coppa: 0, + expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + COPPAEnforced: false, + }, + }, + } + + for _, test := range testCases { + req := newBidRequest(t) + req.Regs = &openrtb.Regs{COPPA: test.coppa} + + results, _, privacyLabels, errs := cleanOpenRTBRequests(context.Background(), req, nil, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{personalInfoAllowed: true}, true, config.Privacy{}, &config.Account{}) + result := results["appnexus"] + + assert.Nil(t, errs) + if test.expectDataScrub { + assert.Equal(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") + assert.Equal(t, result.User.Yob, int64(0), test.description+":User.Yob") + } else { + assert.NotEqual(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") + assert.NotEqual(t, result.User.Yob, int64(0), test.description+":User.Yob") + } + assert.Equal(t, test.expectPrivacyLabels, privacyLabels, test.description+":PrivacyLabels") + } +} + +func TestCleanOpenRTBRequestsSChain(t *testing.T) { + testCases := []struct { + description string + inExt json.RawMessage + inSourceExt json.RawMessage + outSourceExt json.RawMessage + outRequestExt json.RawMessage + hasError bool + }{ + { + description: "Empty root ext and source ext, nil unmarshaled ext", + inExt: nil, + inSourceExt: json.RawMessage(``), + outSourceExt: json.RawMessage(``), + outRequestExt: json.RawMessage(``), + hasError: false, + }, + { + description: "Empty root ext, source ext, and unmarshaled ext", + inExt: json.RawMessage(``), + inSourceExt: json.RawMessage(``), + outSourceExt: json.RawMessage(``), + outRequestExt: json.RawMessage(``), + hasError: false, + }, + { + description: "No schains in root ext and empty source ext. Unmarshaled ext is equivalent to root ext", + inSourceExt: json.RawMessage(``), + inExt: json.RawMessage(`{"prebid":{"schains":[]}}`), + outSourceExt: json.RawMessage(``), + outRequestExt: json.RawMessage(`{"prebid":{}}`), + hasError: false, + }, + { + description: "Use source schain -- no bidder schain or wildcard schain in ext.prebid.schains. Unmarshaled ext is equivalent to root ext", + inSourceExt: json.RawMessage(`{"schain":{"complete":1,"nodes":[{"asi":"example.com","sid":"example1","rid":"ExampleReq1","hp":1}],"ver":"1.0"}}`), + inExt: json.RawMessage(`{"prebid":{"schains":[{"bidders":["bidder1"],"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}]}}`), + outSourceExt: json.RawMessage(`{"schain":{"complete":1,"nodes":[{"asi":"example.com","sid":"example1","rid":"ExampleReq1","hp":1}],"ver":"1.0"}}`), + outRequestExt: json.RawMessage(`{"prebid":{}}`), + hasError: false, + }, + { + description: "Use schain for bidder in ext.prebid.schains. Unmarshaled ext is equivalent to root ext", + inSourceExt: json.RawMessage(``), + inExt: json.RawMessage(`{"prebid":{"schains":[{"bidders":["appnexus"],"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}]}}`), + outSourceExt: json.RawMessage(`{"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}`), + outRequestExt: json.RawMessage(`{"prebid":{}}`), + hasError: false, + }, + { + description: "Use wildcard schain in ext.prebid.schains. Unmarshaled ext is equivalent to root ext", + inSourceExt: json.RawMessage(``), + inExt: json.RawMessage(`{"prebid":{"schains":[{"bidders":["*"],"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}]}}`), + outSourceExt: json.RawMessage(`{"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}`), + outRequestExt: json.RawMessage(`{"prebid":{}}`), + hasError: false, + }, + { + description: "Use schain for bidder in ext.prebid.schains instead of wildcard. Unmarshaled ext is equivalent to root ext", + inSourceExt: json.RawMessage(``), + inExt: json.RawMessage(`{"prebid":{"aliases":{"appnexus":"alias1"},"schains":[{"bidders":["appnexus"],"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}, {"bidders":["*"],"schain":{"complete":1,"nodes":[{"asi":"wildcard.com","sid":"wildcard1","rid":"WildcardReq1","hp":1}],"ver":"1.0"}} ]}}`), + outSourceExt: json.RawMessage(`{"schain":{"complete":1,"nodes":[{"asi":"directseller.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}`), + outRequestExt: json.RawMessage(`{"prebid":{"aliases":{"appnexus":"alias1"}}}`), + hasError: false, + }, + { + description: "Use source schain -- multiple (two) bidder schains in ext.prebid.schains. Unmarshaled ext is equivalent to root ext", + inSourceExt: json.RawMessage(`{"schain":{"complete":1,"nodes":[{"asi":"example.com","sid":"example1","rid":"ExampleReq1","hp":1}],"ver":"1.0"}}`), + inExt: json.RawMessage(`{"prebid":{"schains":[{"bidders":["appnexus"],"schain":{"complete":1,"nodes":[{"asi":"directseller1.com","sid":"00001","rid":"BidRequest1","hp":1}],"ver":"1.0"}}, {"bidders":["appnexus"],"schain":{"complete":1,"nodes":[{"asi":"directseller2.com","sid":"00002","rid":"BidRequest2","hp":1}],"ver":"1.0"}}]}}`), + outSourceExt: nil, + outRequestExt: nil, + hasError: true, + }, + } + + for _, test := range testCases { + req := newBidRequest(t) + req.Source.Ext = test.inSourceExt + + var extRequest *openrtb_ext.ExtRequest + if test.inExt != nil { + req.Ext = test.inExt + unmarshaledExt, err := extractBidRequestExt(req) + assert.NoErrorf(t, err, test.description+":Error unmarshaling inExt") + extRequest = unmarshaledExt + } + + results, _, _, errs := cleanOpenRTBRequests(context.Background(), req, extRequest, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{}, true, config.Privacy{}, &config.Account{}) + result := results["appnexus"] + + if test.hasError == true { + assert.NotNil(t, errs) + assert.Nil(t, result) + } else { + assert.Nil(t, errs) + assert.Equal(t, test.outSourceExt, result.Source.Ext, test.description+":Source.Ext") + assert.Equal(t, test.outRequestExt, result.Ext, test.description+":Ext") + } + } +} + +func TestExtractBidRequestExt(t *testing.T) { + var boolFalse, boolTrue *bool = new(bool), new(bool) + *boolFalse = false + *boolTrue = true + + testCases := []struct { + desc string + inBidRequest *openrtb.BidRequest + outRequestExt *openrtb_ext.ExtRequest + outError error + }{ + { + desc: "Valid vastxml.returnCreative set to false", + inBidRequest: &openrtb.BidRequest{Ext: json.RawMessage(`{"prebid":{"debug":true,"cache":{"vastxml":{"returnCreative":false}}}}`)}, + outRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Debug: true, + Cache: &openrtb_ext.ExtRequestPrebidCache{ + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ + ReturnCreative: boolFalse, + }, + }, + }, + }, + outError: nil, + }, + { + desc: "Valid vastxml.returnCreative set to true", + inBidRequest: &openrtb.BidRequest{Ext: json.RawMessage(`{"prebid":{"debug":true,"cache":{"vastxml":{"returnCreative":true}}}}`)}, + outRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Debug: true, + Cache: &openrtb_ext.ExtRequestPrebidCache{ + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ + ReturnCreative: boolTrue, + }, + }, + }, + }, + outError: nil, + }, + { + desc: "bidRequest nil, we expect an error", + inBidRequest: nil, + outRequestExt: &openrtb_ext.ExtRequest{}, + outError: fmt.Errorf("Error bidRequest should not be nil"), + }, + { + desc: "Non-nil bidRequest with empty Ext, we expect a blank requestExt", + inBidRequest: &openrtb.BidRequest{}, + outRequestExt: &openrtb_ext.ExtRequest{}, + outError: nil, + }, + { + desc: "Non-nil bidRequest with non-empty, invalid Ext, we expect unmarshaling error", + inBidRequest: &openrtb.BidRequest{Ext: json.RawMessage(`invalid`)}, + outRequestExt: &openrtb_ext.ExtRequest{}, + outError: fmt.Errorf("Error decoding Request.ext : invalid character 'i' looking for beginning of value"), + }, + } + for _, test := range testCases { + actualRequestExt, actualErr := extractBidRequestExt(test.inBidRequest) + + // Given that assert.Equal asserts pointer variable equality based on the equality of the referenced values (as opposed to + // the memory addresses) the call below asserts whether or not *test.outRequestExt.Prebid.Cache.VastXML.ReturnCreative boolean + // value is equal to that of *actualRequestExt.Prebid.Cache.VastXML.ReturnCreative + assert.Equal(t, test.outRequestExt, actualRequestExt, "%s. Unexpected RequestExt value. \n", test.desc) + assert.Equal(t, test.outError, actualErr, "%s. Unexpected error value. \n", test.desc) + } +} + +func TestGetExtCacheInstructions(t *testing.T) { + var boolFalse, boolTrue *bool = new(bool), new(bool) + *boolFalse = false + *boolTrue = true + + testCases := []struct { + desc string + inRequestExt *openrtb_ext.ExtRequest + outCacheInstructions extCacheInstructions + }{ + { + desc: "Nil inRequestExt, all cache flags false except for returnCreative that defaults to true", + inRequestExt: nil, + outCacheInstructions: extCacheInstructions{ + cacheBids: false, + cacheVAST: false, + returnCreative: true, + }, + }, + { + desc: "Non-nil inRequestExt, nil Cache field, all cache flags false except for returnCreative that defaults to true", + inRequestExt: &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{Cache: nil}}, + outCacheInstructions: extCacheInstructions{ + cacheBids: false, + cacheVAST: false, + returnCreative: true, + }, + }, + { + desc: "Non-nil Cache field, both ExtRequestPrebidCacheBids and ExtRequestPrebidCacheVAST nil returnCreative that defaults to true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: nil, + VastXML: nil, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: false, + cacheVAST: false, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST with unspecified ReturnCreative field, cacheVAST = true and returnCreative defaults to true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: nil, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: false, + cacheVAST: true, + returnCreative: true, // default value + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST where ReturnCreative is set to false, cacheVAST = true and returnCreative = false", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: nil, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ReturnCreative: boolFalse}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: false, + cacheVAST: true, + returnCreative: false, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST where ReturnCreative is set to true, cacheVAST = true and returnCreative = true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: nil, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ReturnCreative: boolTrue}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: false, + cacheVAST: true, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheBids with unspecified ReturnCreative field, cacheBids = true and returnCreative defaults to true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{}, + VastXML: nil, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: false, + returnCreative: true, // default value + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheBids where ReturnCreative is set to false, cacheBids = true and returnCreative = false", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{ReturnCreative: boolFalse}, + VastXML: nil, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: false, + returnCreative: false, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheBids where ReturnCreative is set to true, cacheBids = true and returnCreative = true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{ReturnCreative: boolTrue}, + VastXML: nil, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: false, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheBids and ExtRequest.Cache.ExtRequestPrebidCacheVAST, neither specify a ReturnCreative field value, all extCacheInstructions fields set to true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheBids and ExtRequest.Cache.ExtRequestPrebidCacheVAST sets ReturnCreative to true, all extCacheInstructions fields set to true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ReturnCreative: boolTrue}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheBids and ExtRequest.Cache.ExtRequestPrebidCacheVAST sets ReturnCreative to false, returnCreative = false", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ReturnCreative: boolFalse}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: false, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST and ExtRequest.Cache.ExtRequestPrebidCacheBids sets ReturnCreative to true, all extCacheInstructions fields set to true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{ReturnCreative: boolTrue}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST and ExtRequest.Cache.ExtRequestPrebidCacheBids sets ReturnCreative to false, returnCreative = false", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{ReturnCreative: boolFalse}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: false, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST and ExtRequest.Cache.ExtRequestPrebidCacheBids set different ReturnCreative values, returnCreative = true because one of them is true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{ReturnCreative: boolFalse}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ReturnCreative: boolTrue}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: true, + }, + }, + { + desc: "Non-nil ExtRequest.Cache.ExtRequestPrebidCacheVAST and ExtRequest.Cache.ExtRequestPrebidCacheBids set different ReturnCreative values, returnCreative = true because one of them is true", + inRequestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Cache: &openrtb_ext.ExtRequestPrebidCache{ + Bids: &openrtb_ext.ExtRequestPrebidCacheBids{ReturnCreative: boolTrue}, + VastXML: &openrtb_ext.ExtRequestPrebidCacheVAST{ReturnCreative: boolFalse}, + }, + }, + }, + outCacheInstructions: extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + returnCreative: true, + }, + }, + } + + for _, test := range testCases { + cacheInstructions := getExtCacheInstructions(test.inRequestExt) + + assert.Equal(t, test.outCacheInstructions.cacheBids, cacheInstructions.cacheBids, "%s. Unexpected shouldCacheBids value. \n", test.desc) + assert.Equal(t, test.outCacheInstructions.cacheVAST, cacheInstructions.cacheVAST, "%s. Unexpected shouldCacheVAST value. \n", test.desc) + assert.Equal(t, test.outCacheInstructions.returnCreative, cacheInstructions.returnCreative, "%s. Unexpected returnCreative value. \n", test.desc) + } +} + +func TestGetExtTargetData(t *testing.T) { + type inTest struct { + requestExt *openrtb_ext.ExtRequest + cacheInstructions *extCacheInstructions + } + type outTest struct { + targetData *targetData + nilTargetData bool + } + testCases := []struct { + desc string + in inTest + out outTest + }{ + { + "nil requestExt, nil outTargetData", + inTest{ + requestExt: nil, + cacheInstructions: &extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + }, + }, + outTest{targetData: nil, nilTargetData: true}, + }, + { + "Valid requestExt, nil Targeting field, nil outTargetData", + inTest{ + requestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Targeting: nil, + }, + }, + cacheInstructions: &extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + }, + }, + outTest{targetData: nil, nilTargetData: true}, + }, + { + "Valid targeting data in requestExt, valid outTargetData", + inTest{ + requestExt: &openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + Targeting: &openrtb_ext.ExtRequestTargeting{ + PriceGranularity: openrtb_ext.PriceGranularity{ + Precision: 2, + Ranges: []openrtb_ext.GranularityRange{{Min: 0.00, Max: 5.00, Increment: 1.00}}, + }, + IncludeWinners: true, + IncludeBidderKeys: true, + }, + }, + }, + cacheInstructions: &extCacheInstructions{ + cacheBids: true, + cacheVAST: true, + }, + }, + outTest{ + targetData: &targetData{ + priceGranularity: openrtb_ext.PriceGranularity{ + Precision: 2, + Ranges: []openrtb_ext.GranularityRange{{Min: 0.00, Max: 5.00, Increment: 1.00}}, + }, + includeWinners: true, + includeBidderKeys: true, + includeCacheBids: true, + includeCacheVast: true, + }, + nilTargetData: false, + }, + }, + } + for _, test := range testCases { + actualTargetData := getExtTargetData(test.in.requestExt, test.in.cacheInstructions) + + if test.out.nilTargetData { + assert.Nil(t, actualTargetData, "%s. Targeting data should be nil. \n", test.desc) + } else { + assert.NotNil(t, actualTargetData, "%s. Targeting data should NOT be nil. \n", test.desc) + assert.Equal(t, *test.out.targetData, *actualTargetData, "%s. Unexpected targeting data value. \n", test.desc) + } + } +} + +func TestGetDebugInfo(t *testing.T) { + type inTest struct { + bidRequest *openrtb.BidRequest + requestExt *openrtb_ext.ExtRequest + } + testCases := []struct { + desc string + in inTest + out bool + }{ + { + desc: "Nil bid request, nil requestExt", + in: inTest{nil, nil}, + out: false, + }, + { + desc: "bid request test == 0, nil requestExt", + in: inTest{&openrtb.BidRequest{Test: 0}, nil}, + out: false, + }, + { + desc: "Nil bid request, requestExt debug flag false", + in: inTest{nil, &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{Debug: false}}}, + out: false, + }, + { + desc: "bid request test == 0, requestExt debug flag false", + in: inTest{&openrtb.BidRequest{Test: 0}, &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{Debug: false}}}, + out: false, + }, + { + desc: "bid request test == 1, requestExt debug flag false", + in: inTest{&openrtb.BidRequest{Test: 1}, &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{Debug: false}}}, + out: true, + }, + { + desc: "bid request test == 0, requestExt debug flag true", + in: inTest{&openrtb.BidRequest{Test: 0}, &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{Debug: true}}}, + out: true, + }, + { + desc: "bid request test == 1, requestExt debug flag true", + in: inTest{&openrtb.BidRequest{Test: 1}, &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{Debug: true}}}, + out: true, + }, + } + for _, test := range testCases { + actualDebugInfo := getDebugInfo(test.in.bidRequest, test.in.requestExt) + + assert.Equal(t, test.out, actualDebugInfo, "%s. Unexpected debug value. \n", test.desc) + } +} + +func TestGetExtBidAdjustmentFactors(t *testing.T) { + testCases := []struct { + desc string + inRequestExt *openrtb_ext.ExtRequest + outBidAdjustmentFactors map[string]float64 + }{ + { + desc: "Nil request ext", + inRequestExt: nil, + outBidAdjustmentFactors: nil, + }, + { + desc: "Non-nil request ext, nil BidAdjustmentFactors field", + inRequestExt: &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{BidAdjustmentFactors: nil}}, + outBidAdjustmentFactors: nil, + }, + { + desc: "Non-nil request ext, valid BidAdjustmentFactors field", + inRequestExt: &openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{BidAdjustmentFactors: map[string]float64{"bid-factor": 1.0}}}, + outBidAdjustmentFactors: map[string]float64{"bid-factor": 1.0}, + }, + } + for _, test := range testCases { + actualBidAdjustmentFactors := getExtBidAdjustmentFactors(test.inRequestExt) + + assert.Equal(t, test.outBidAdjustmentFactors, actualBidAdjustmentFactors, "%s. Unexpected BidAdjustmentFactors value. \n", test.desc) } } @@ -141,34 +947,47 @@ func TestCleanOpenRTBRequestsLMT(t *testing.T) { disabled int8 = 0 ) testCases := []struct { - description string - lmt *int8 - enforceLMT bool - expectDataScrub bool + description string + lmt *int8 + enforceLMT bool + expectDataScrub bool + expectPrivacyLabels pbsmetrics.PrivacyLabels }{ { description: "Feature Flag Enabled - OpenTRB Enabled", lmt: &enabled, enforceLMT: true, expectDataScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + LMTEnforced: true, + }, }, { description: "Feature Flag Disabled - OpenTRB Enabled", lmt: &enabled, enforceLMT: false, expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + LMTEnforced: false, + }, }, { description: "Feature Flag Enabled - OpenTRB Disabled", lmt: &disabled, enforceLMT: true, expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + LMTEnforced: false, + }, }, { description: "Feature Flag Disabled - OpenTRB Disabled", lmt: &disabled, enforceLMT: false, expectDataScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + LMTEnforced: false, + }, }, } @@ -182,11 +1001,10 @@ func TestCleanOpenRTBRequestsLMT(t *testing.T) { }, } - results, _, errs := cleanOpenRTBRequests(context.Background(), req, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{}, true, privacyConfig) + results, _, privacyLabels, errs := cleanOpenRTBRequests(context.Background(), req, nil, &emptyUsersync{}, map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, pbsmetrics.Labels{}, &permissionsMock{personalInfoAllowed: true}, true, privacyConfig, &config.Account{}) result := results["appnexus"] assert.Nil(t, errs) - if test.expectDataScrub { assert.Equal(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") assert.Equal(t, result.Device.DIDMD5, "", test.description+":Device.DIDMD5") @@ -194,6 +1012,164 @@ func TestCleanOpenRTBRequestsLMT(t *testing.T) { assert.NotEqual(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") assert.NotEqual(t, result.Device.DIDMD5, "", test.description+":Device.DIDMD5") } + assert.Equal(t, test.expectPrivacyLabels, privacyLabels, test.description+":PrivacyLabels") + } +} + +func TestCleanOpenRTBRequestsGDPR(t *testing.T) { + trueValue, falseValue := true, false + + testCases := []struct { + description string + gdprAccountEnabled *bool + gdprHostEnabled bool + gdpr string + gdprConsent string + gdprScrub bool + expectPrivacyLabels pbsmetrics.PrivacyLabels + }{ + { + description: "Enforce - TCF Invalid", + gdprAccountEnabled: &trueValue, + gdprHostEnabled: true, + gdpr: "1", + gdprConsent: "malformed", + gdprScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: "", + }, + }, + { + description: "Enforce - TCF 1", + gdprAccountEnabled: &trueValue, + gdprHostEnabled: true, + gdpr: "1", + gdprConsent: "BONV8oqONXwgmADACHENAO7pqzAAppY", + gdprScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV1, + }, + }, + { + description: "Enforce - TCF 2", + gdprAccountEnabled: &trueValue, + gdprHostEnabled: true, + gdpr: "1", + gdprConsent: "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA", + gdprScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV2, + }, + }, + { + description: "Not Enforce - TCF 1", + gdprAccountEnabled: &trueValue, + gdprHostEnabled: true, + gdpr: "0", + gdprConsent: "BONV8oqONXwgmADACHENAO7pqzAAppY", + gdprScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: false, + GDPRTCFVersion: "", + }, + }, + { + description: "Enforce - TCF 1; account GDPR enabled, host GDPR setting disregarded", + gdprAccountEnabled: &trueValue, + gdprHostEnabled: false, + gdpr: "1", + gdprConsent: "BONV8oqONXwgmADACHENAO7pqzAAppY", + gdprScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV1, + }, + }, + { + description: "Not Enforce - TCF 1; account GDPR disabled, host GDPR setting disregarded", + gdprAccountEnabled: &falseValue, + gdprHostEnabled: true, + gdpr: "1", + gdprConsent: "BONV8oqONXwgmADACHENAO7pqzAAppY", + gdprScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: false, + GDPRTCFVersion: "", + }, + }, + { + description: "Enforce - TCF 1; account GDPR not specified, host GDPR enabled", + gdprAccountEnabled: nil, + gdprHostEnabled: true, + gdpr: "1", + gdprConsent: "BONV8oqONXwgmADACHENAO7pqzAAppY", + gdprScrub: true, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV1, + }, + }, + { + description: "Not Enforce - TCF 1; account GDPR not specified, host GDPR disabled", + gdprAccountEnabled: nil, + gdprHostEnabled: false, + gdpr: "1", + gdprConsent: "BONV8oqONXwgmADACHENAO7pqzAAppY", + gdprScrub: false, + expectPrivacyLabels: pbsmetrics.PrivacyLabels{ + GDPREnforced: false, + GDPRTCFVersion: "", + }, + }, + } + + for _, test := range testCases { + req := newBidRequest(t) + req.User.Ext = json.RawMessage(`{"consent":"` + test.gdprConsent + `"}`) + req.Regs = &openrtb.Regs{ + Ext: json.RawMessage(`{"gdpr":` + test.gdpr + `}`), + } + + privacyConfig := config.Privacy{ + GDPR: config.GDPR{ + Enabled: test.gdprHostEnabled, + TCF2: config.TCF2{ + Enabled: true, + }, + }, + } + + accountConfig := config.Account{ + GDPR: config.AccountGDPR{ + Enabled: test.gdprAccountEnabled, + }, + } + + results, _, privacyLabels, errs := cleanOpenRTBRequests( + context.Background(), + req, + nil, + &emptyUsersync{}, + map[openrtb_ext.BidderName]*pbsmetrics.AdapterLabels{}, + pbsmetrics.Labels{}, + &permissionsMock{personalInfoAllowed: !test.gdprScrub}, + true, + privacyConfig, + &accountConfig) + result := results["appnexus"] + + assert.Nil(t, errs) + if test.gdprScrub { + assert.Equal(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") + assert.Equal(t, result.Device.DIDMD5, "", test.description+":Device.DIDMD5") + } else { + assert.NotEqual(t, result.User.BuyerUID, "", test.description+":User.BuyerUID") + assert.NotEqual(t, result.Device.DIDMD5, "", test.description+":Device.DIDMD5") + } + assert.Equal(t, test.expectPrivacyLabels, privacyLabels, test.description+":PrivacyLabels") } } @@ -266,6 +1242,7 @@ func newBidRequest(t *testing.T) *openrtb.BidRequest { User: &openrtb.User{ ID: "our-id", BuyerUID: "their-id", + Yob: 1982, Ext: json.RawMessage(`{"digitrust":{"id":"digi-id","keyv":1,"pref":1}}`), }, Imp: []openrtb.Imp{{ @@ -302,5 +1279,99 @@ func TestRandomizeList(t *testing.T) { if len(adapters) != 1 { t.Errorf("RandomizeList, expected a list of 1, found %d", len(adapters)) } +} + +func TestBidderToPrebidChains(t *testing.T) { + input := openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + SChains: []*openrtb_ext.ExtRequestPrebidSChain{ + { + Bidders: []string{"Bidder1", "Bidder2"}, + SChain: openrtb_ext.ExtRequestPrebidSChainSChain{ + Complete: 1, + Nodes: []*openrtb_ext.ExtRequestPrebidSChainSChainNode{ + { + ASI: "asi1", + SID: "sid1", + Name: "name1", + RID: "rid1", + Domain: "domain1", + HP: 1, + }, + { + ASI: "asi2", + SID: "sid2", + Name: "name2", + RID: "rid2", + Domain: "domain2", + HP: 2, + }, + }, + Ver: "version1", + }, + }, + { + Bidders: []string{"Bidder3", "Bidder4"}, + SChain: openrtb_ext.ExtRequestPrebidSChainSChain{}, + }, + }, + }, + } + + output, err := BidderToPrebidSChains(&input) + + assert.Nil(t, err) + assert.Equal(t, len(output), 4) + assert.Same(t, output["Bidder1"], &input.Prebid.SChains[0].SChain) + assert.Same(t, output["Bidder2"], &input.Prebid.SChains[0].SChain) + assert.Same(t, output["Bidder3"], &input.Prebid.SChains[1].SChain) + assert.Same(t, output["Bidder4"], &input.Prebid.SChains[1].SChain) +} + +func TestBidderToPrebidChainsDiscardMultipleChainsForBidder(t *testing.T) { + input := openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + SChains: []*openrtb_ext.ExtRequestPrebidSChain{ + { + Bidders: []string{"Bidder1"}, + SChain: openrtb_ext.ExtRequestPrebidSChainSChain{}, + }, + { + Bidders: []string{"Bidder1", "Bidder2"}, + SChain: openrtb_ext.ExtRequestPrebidSChainSChain{}, + }, + }, + }, + } + + output, err := BidderToPrebidSChains(&input) + + assert.NotNil(t, err) + assert.Nil(t, output) +} + +func TestBidderToPrebidChainsNilSChains(t *testing.T) { + input := openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + SChains: nil, + }, + } + + output, err := BidderToPrebidSChains(&input) + + assert.Nil(t, err) + assert.Equal(t, len(output), 0) +} + +func TestBidderToPrebidChainsZeroLengthSChains(t *testing.T) { + input := openrtb_ext.ExtRequest{ + Prebid: openrtb_ext.ExtRequestPrebid{ + SChains: []*openrtb_ext.ExtRequestPrebidSChain{}, + }, + } + + output, err := BidderToPrebidSChains(&input) + assert.Nil(t, err) + assert.Equal(t, len(output), 0) } diff --git a/gdpr/gdpr.go b/gdpr/gdpr.go index b4cb336986a..eab8e4bd8d1 100644 --- a/gdpr/gdpr.go +++ b/gdpr/gdpr.go @@ -23,15 +23,16 @@ type Permissions interface { // Determines whether or not to send PI information to a bidder, or mask it out. // // If the consent string was nonsensical, the returned error will be an ErrorMalformedConsent. - PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) + PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) // Exposes the AMP execption flag AMPException() bool } +// Versions of the GDPR TCF technical specification. const ( - tCF1 uint8 = 1 - tCF2 uint8 = 2 + tcf1SpecVersion uint8 = 1 + tcf2SpecVersion uint8 = 2 ) // NewPermissions gets an instance of the Permissions for use elsewhere in the project. @@ -45,8 +46,8 @@ func NewPermissions(ctx context.Context, cfg config.GDPR, vendorIDs map[openrtb_ cfg: cfg, vendorIDs: vendorIDs, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: newVendorListFetcher(ctx, cfg, client, vendorListURLMaker, tCF1), - tCF2: newVendorListFetcher(ctx, cfg, client, vendorListURLMaker, tCF2)}, + tcf1SpecVersion: newVendorListFetcher(ctx, cfg, client, vendorListURLMaker, tcf1SpecVersion), + tcf2SpecVersion: newVendorListFetcher(ctx, cfg, client, vendorListURLMaker, tcf2SpecVersion)}, } } diff --git a/gdpr/impl.go b/gdpr/impl.go index 7fa6fde588f..8fe8908ab40 100644 --- a/gdpr/impl.go +++ b/gdpr/impl.go @@ -42,10 +42,10 @@ func (p *permissionsImpl) BidderSyncAllowed(ctx context.Context, bidder openrtb_ return false, nil } -func (p *permissionsImpl) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) { +func (p *permissionsImpl) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { _, ok := p.cfg.NonStandardPublisherMap[PublisherID] if ok { - return true, true, nil + return true, true, true, nil } id, ok := p.vendorIDs[bidder] @@ -54,10 +54,10 @@ func (p *permissionsImpl) PersonalInfoAllowed(ctx context.Context, bidder openrt } if consent == "" { - return p.cfg.UsersyncIfAmbiguous, p.cfg.UsersyncIfAmbiguous, nil + return p.cfg.UsersyncIfAmbiguous, p.cfg.UsersyncIfAmbiguous, p.cfg.UsersyncIfAmbiguous, nil } - return false, false, nil + return false, false, false, nil } func (p *permissionsImpl) AMPException() bool { @@ -98,19 +98,19 @@ func (p *permissionsImpl) allowSync(ctx context.Context, vendorID uint16, consen return false, nil } -func (p *permissionsImpl) allowPI(ctx context.Context, vendorID uint16, consent string) (bool, bool, error) { +func (p *permissionsImpl) allowPI(ctx context.Context, vendorID uint16, consent string) (bool, bool, bool, error) { // If we're not given a consent string, respect the preferences in the app config. if consent == "" { - return p.cfg.UsersyncIfAmbiguous, p.cfg.UsersyncIfAmbiguous, nil + return p.cfg.UsersyncIfAmbiguous, p.cfg.UsersyncIfAmbiguous, p.cfg.UsersyncIfAmbiguous, nil } parsedConsent, vendor, err := p.parseVendor(ctx, vendorID, consent) if err != nil { - return false, false, err + return false, false, false, err } if vendor == nil { - return false, false, nil + return false, false, false, nil } if parsedConsent.Version() == 2 { @@ -118,21 +118,22 @@ func (p *permissionsImpl) allowPI(ctx context.Context, vendorID uint16, consent return p.allowPITCF2(parsedConsent, vendor, vendorID) } if (vendor.Purpose(consentconstants.InfoStorageAccess) || vendor.LegitimateInterest(consentconstants.InfoStorageAccess)) && parsedConsent.PurposeAllowed(consentconstants.InfoStorageAccess) && (vendor.Purpose(consentconstants.PersonalizationProfile) || vendor.LegitimateInterest(consentconstants.PersonalizationProfile)) && parsedConsent.PurposeAllowed(consentconstants.PersonalizationProfile) && parsedConsent.VendorConsent(vendorID) { - return true, true, nil + return true, true, true, nil } } else { if (vendor.Purpose(tcf1constants.InfoStorageAccess) || vendor.LegitimateInterest(tcf1constants.InfoStorageAccess)) && parsedConsent.PurposeAllowed(tcf1constants.InfoStorageAccess) && (vendor.Purpose(tcf1constants.AdSelectionDeliveryReporting) || vendor.LegitimateInterest(tcf1constants.AdSelectionDeliveryReporting)) && parsedConsent.PurposeAllowed(tcf1constants.AdSelectionDeliveryReporting) && parsedConsent.VendorConsent(vendorID) { - return true, true, nil + return true, true, true, nil } } - return false, false, nil + return false, false, false, nil } -func (p *permissionsImpl) allowPITCF2(parsedConsent api.VendorConsents, vendor api.Vendor, vendorID uint16) (allowPI bool, allowGeo bool, err error) { +func (p *permissionsImpl) allowPITCF2(parsedConsent api.VendorConsents, vendor api.Vendor, vendorID uint16) (allowPI bool, allowGeo bool, allowID bool, err error) { consent, ok := parsedConsent.(tcf2.ConsentMetadata) err = nil allowPI = false allowGeo = false + allowID = false if !ok { err = fmt.Errorf("Unable to access TCF2 parsed consent") return @@ -142,6 +143,12 @@ func (p *permissionsImpl) allowPITCF2(parsedConsent api.VendorConsents, vendor a } else { allowGeo = true } + for i := 2; i <= 10; i++ { + if p.checkPurpose(consent, vendor, vendorID, tcf1constants.Purpose(i)) { + allowID = true + break + } + } // Set to true so any purpose check can flip it to false allowPI = true if p.cfg.TCF2.Purpose1.Enabled { @@ -214,10 +221,29 @@ func (a AlwaysAllow) BidderSyncAllowed(ctx context.Context, bidder openrtb_ext.B return true, nil } -func (a AlwaysAllow) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, error) { - return true, true, nil +func (a AlwaysAllow) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { + return true, true, true, nil } func (a AlwaysAllow) AMPException() bool { return false } + +// Exporting to allow for easy test setups +type AlwaysFail struct{} + +func (a AlwaysFail) HostCookiesAllowed(ctx context.Context, consent string) (bool, error) { + return false, nil +} + +func (a AlwaysFail) BidderSyncAllowed(ctx context.Context, bidder openrtb_ext.BidderName, consent string) (bool, error) { + return false, nil +} + +func (a AlwaysFail) PersonalInfoAllowed(ctx context.Context, bidder openrtb_ext.BidderName, PublisherID string, consent string) (bool, bool, bool, error) { + return false, false, false, nil +} + +func (a AlwaysFail) AMPException() bool { + return false +} diff --git a/gdpr/impl_test.go b/gdpr/impl_test.go index 0635ee4e512..b65e9cf4824 100644 --- a/gdpr/impl_test.go +++ b/gdpr/impl_test.go @@ -23,8 +23,8 @@ func TestNoConsentButAllowByDefault(t *testing.T) { }, vendorIDs: nil, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: failedListFetcher, - tCF2: failedListFetcher, + tcf1SpecVersion: failedListFetcher, + tcf2SpecVersion: failedListFetcher, }, } allowSync, err := perms.BidderSyncAllowed(context.Background(), openrtb_ext.BidderAppnexus, "") @@ -43,8 +43,8 @@ func TestNoConsentAndRejectByDefault(t *testing.T) { }, vendorIDs: nil, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: failedListFetcher, - tCF2: failedListFetcher, + tcf1SpecVersion: failedListFetcher, + tcf2SpecVersion: failedListFetcher, }, } allowSync, err := perms.BidderSyncAllowed(context.Background(), openrtb_ext.BidderAppnexus, "") @@ -56,12 +56,11 @@ func TestNoConsentAndRejectByDefault(t *testing.T) { } func TestAllowedSyncs(t *testing.T) { - vendorListData := mockVendorListData(t, 1, map[uint16]*purposes{ - 2: { - purposes: []int{1}, - }, - 3: { - purposes: []int{1}, + vendorListData := tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 1, + Vendors: []tcf1Vendor{ + {ID: 2, Purposes: []int{1}}, + {ID: 3, Purposes: []int{1}}, }, }) perms := permissionsImpl{ @@ -73,10 +72,10 @@ func TestAllowedSyncs(t *testing.T) { openrtb_ext.BidderPubmatic: 3, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), }, @@ -92,12 +91,11 @@ func TestAllowedSyncs(t *testing.T) { } func TestProhibitedPurposes(t *testing.T) { - vendorListData := mockVendorListData(t, 1, map[uint16]*purposes{ - 2: { - purposes: []int{1}, // cookie reads/writes - }, - 3: { - purposes: []int{3}, // ad personalization + vendorListData := tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 1, + Vendors: []tcf1Vendor{ + {ID: 2, Purposes: []int{1}}, // cookie reads/writes + {ID: 3, Purposes: []int{3}}, // ad personalization }, }) perms := permissionsImpl{ @@ -109,10 +107,10 @@ func TestProhibitedPurposes(t *testing.T) { openrtb_ext.BidderPubmatic: 3, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), }, @@ -128,12 +126,11 @@ func TestProhibitedPurposes(t *testing.T) { } func TestProhibitedVendors(t *testing.T) { - vendorListData := mockVendorListData(t, 1, map[uint16]*purposes{ - 2: { - purposes: []int{1}, // cookie reads/writes - }, - 3: { - purposes: []int{3}, // ad personalization + vendorListData := tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 1, + Vendors: []tcf1Vendor{ + {ID: 2, Purposes: []int{1}}, // cookie reads/writes + {ID: 3, Purposes: []int{3}}, // ad personalization }, }) perms := permissionsImpl{ @@ -145,10 +142,10 @@ func TestProhibitedVendors(t *testing.T) { openrtb_ext.BidderPubmatic: 3, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), }, @@ -169,8 +166,8 @@ func TestMalformedConsent(t *testing.T) { HostVendorID: 2, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: listFetcher(nil), - tCF2: listFetcher(nil), + tcf1SpecVersion: listFetcher(nil), + tcf2SpecVersion: listFetcher(nil), }, } @@ -180,12 +177,11 @@ func TestMalformedConsent(t *testing.T) { } func TestAllowPersonalInfo(t *testing.T) { - vendorListData := mockVendorListData(t, 1, map[uint16]*purposes{ - 2: { - purposes: []int{1}, // cookie reads/writes - }, - 3: { - purposes: []int{1, 3}, // ad personalization + vendorListData := tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 1, + Vendors: []tcf1Vendor{ + {ID: 2, Purposes: []int{1}}, // cookie reads/writes + {ID: 3, Purposes: []int{1, 3}}, // ad personalization }, }) perms := permissionsImpl{ @@ -197,49 +193,64 @@ func TestAllowPersonalInfo(t *testing.T) { openrtb_ext.BidderPubmatic: 3, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 1: parseVendorListData(t, vendorListData), }), }, } // PI needs both purposes to succeed - allowPI, _, err := perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderAppnexus, "", "BOS2bx5OS2bx5ABABBAAABoAAAABBwAA") + allowPI, _, _, err := perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderAppnexus, "", "BOS2bx5OS2bx5ABABBAAABoAAAABBwAA") assertNilErr(t, err) assertBoolsEqual(t, false, allowPI) - allowPI, _, err = perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderPubmatic, "", "BOS2bx5OS2bx5ABABBAAABoAAAABBwAA") + allowPI, _, _, err = perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderPubmatic, "", "BOS2bx5OS2bx5ABABBAAABoAAAABBwAA") assertNilErr(t, err) assertBoolsEqual(t, true, allowPI) // Assert that an item that otherwise would not be allowed PI access, gets approved because it is found in the GDPR.NonStandardPublishers array - perms.cfg.NonStandardPublisherMap = map[string]int{"appNexusAppID": 1} - allowPI, _, err = perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderAppnexus, "appNexusAppID", "BOS2bx5OS2bx5ABABBAAABoAAAABBwAA") + perms.cfg.NonStandardPublisherMap = map[string]struct{}{"appNexusAppID": {}} + allowPI, _, _, err = perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderAppnexus, "appNexusAppID", "BOS2bx5OS2bx5ABABBAAABoAAAABBwAA") assertNilErr(t, err) assertBoolsEqual(t, true, allowPI) } -var tcf2BasicPurposes = map[uint16]*purposes{ - 2: {purposes: []int{1}}, //cookie reads/writes - 6: {purposes: []int{1, 2, 4}}, // ad personalization - 8: {purposes: []int{1, 7}}, - 10: {purposes: []int{2, 4, 7}}, - 32: {purposes: []int{1, 2, 4, 7}}, -} -var tcf2LegitInterests = map[uint16]*purposes{ - 6: {purposes: []int{7}}, - 8: {purposes: []int{2, 4}}, -} -var tcf2SpecialPuproses = map[uint16]*purposes{ - 6: {purposes: []int{1}}, - 10: {purposes: []int{1}}, -} -var tcf2FlexPurposes = map[uint16]*purposes{ - 6: {purposes: []int{1, 2, 4, 7}}, +func buildTCF2VendorList34() tcf2VendorList { + return tcf2VendorList{ + VendorListVersion: 2, + Vendors: map[string]*tcf2Vendor{ + "2": { + ID: 2, + Purposes: []int{1}, + }, + "6": { + ID: 6, + Purposes: []int{1, 2, 4}, + LegIntPurposes: []int{7}, + SpecialPurposes: []int{1}, + FlexiblePurposes: []int{1, 2, 4, 7}, + }, + "8": { + ID: 8, + Purposes: []int{1, 7}, + LegIntPurposes: []int{2, 4}, + }, + "10": { + ID: 10, + Purposes: []int{2, 4, 7}, + SpecialPurposes: []int{1}, + }, + "32": { + ID: 32, + Purposes: []int{1, 2, 4, 7}, + }, + }, + } } + var tcf2Config = config.GDPR{ HostVendorID: 2, TCF2: config.TCF2{ @@ -257,10 +268,11 @@ type tcf2TestDef struct { consent string allowPI bool allowGeo bool + allowID bool } func TestAllowPersonalInfoTCF2(t *testing.T) { - vendorListData := mockVendorListDataTCF2(t, 2, tcf2BasicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -269,14 +281,14 @@ func TestAllowPersonalInfoTCF2(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, } - // COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA : TCF2 with full consensts to purposes and vendors 2, 6, 8 + // COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA : TCF2 with full consents to purposes and vendors 2, 6, 8 // PI needs all purposes to succeed testDefs := []tcf2TestDef{ { @@ -285,6 +297,7 @@ func TestAllowPersonalInfoTCF2(t *testing.T) { consent: "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA", allowPI: false, allowGeo: false, + allowID: false, }, { description: "Pubmatic vendor test, flex purposes claimed", @@ -292,6 +305,7 @@ func TestAllowPersonalInfoTCF2(t *testing.T) { consent: "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA", allowPI: true, allowGeo: true, + allowID: true, }, { description: "Rubicon vendor test, Specific purposes/LIs claimed, no geo claimed", @@ -299,19 +313,21 @@ func TestAllowPersonalInfoTCF2(t *testing.T) { consent: "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA", allowPI: true, allowGeo: false, + allowID: true, }, } for _, td := range testDefs { - allowPI, allowGeo, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) + allowPI, allowGeo, allowID, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) assert.NoErrorf(t, err, "Error processing PersonalInfoAllowed for %s", td.description) assert.EqualValuesf(t, td.allowPI, allowPI, "AllowPI failure on %s", td.description) assert.EqualValuesf(t, td.allowGeo, allowGeo, "AllowGeo failure on %s", td.description) + assert.EqualValuesf(t, td.allowID, allowID, "AllowGeo failure on %s", td.description) } } func TestAllowPersonalInfoWhitelistTCF2(t *testing.T) { - vendorListData := mockVendorListDataTCF2(t, 2, tcf2BasicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -320,23 +336,23 @@ func TestAllowPersonalInfoWhitelistTCF2(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, } // Assert that an item that otherwise would not be allowed PI access, gets approved because it is found in the GDPR.NonStandardPublishers array - perms.cfg.NonStandardPublisherMap = map[string]int{"appNexusAppID": 1} - allowPI, allowGeo, err := perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderAppnexus, "appNexusAppID", "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA") + perms.cfg.NonStandardPublisherMap = map[string]struct{}{"appNexusAppID": {}} + allowPI, allowGeo, allowID, err := perms.PersonalInfoAllowed(context.Background(), openrtb_ext.BidderAppnexus, "appNexusAppID", "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA") assert.NoErrorf(t, err, "Error processing PersonalInfoAllowed") assert.EqualValuesf(t, true, allowPI, "AllowPI failure") assert.EqualValuesf(t, true, allowGeo, "AllowGeo failure") - + assert.EqualValuesf(t, true, allowID, "AllowID failure") } func TestAllowPersonalInfoTCF2PubRestrict(t *testing.T) { - vendorListData := mockVendorListDataTCF2(t, 2, tcf2BasicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -345,8 +361,8 @@ func TestAllowPersonalInfoTCF2PubRestrict(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 15: parseVendorListDataV2(t, vendorListData), }), }, @@ -361,6 +377,7 @@ func TestAllowPersonalInfoTCF2PubRestrict(t *testing.T) { consent: "COwAdDhOwAdDhN4ABAENAPCgAAQAAv___wAAAFP_AAp_4AI6ACACAA", allowPI: false, allowGeo: false, + allowID: false, }, { description: "Pubmatic vendor test, flex purposes claimed", @@ -368,6 +385,7 @@ func TestAllowPersonalInfoTCF2PubRestrict(t *testing.T) { consent: "COwAdDhOwAdDhN4ABAENAPCgAAQAAv___wAAAFP_AAp_4AI6ACACAA", allowPI: false, allowGeo: false, + allowID: false, }, { description: "Rubicon vendor test, Specific purposes/LIs claimed, no geo claimed", @@ -375,19 +393,21 @@ func TestAllowPersonalInfoTCF2PubRestrict(t *testing.T) { consent: "COwAdDhOwAdDhN4ABAENAPCgAAQAAv___wAAAFP_AAp_4AI6ACACAA", allowPI: false, allowGeo: false, + allowID: true, }, } for _, td := range testDefs { - allowPI, allowGeo, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) + allowPI, allowGeo, allowID, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) assert.NoErrorf(t, err, "Error processing PersonalInfoAllowed for %s", td.description) assert.EqualValuesf(t, td.allowPI, allowPI, "AllowPI failure on %s", td.description) assert.EqualValuesf(t, td.allowGeo, allowGeo, "AllowGeo failure on %s", td.description) + assert.EqualValuesf(t, td.allowID, allowID, "AllowPI failure on %s", td.description) } } func TestAllowPersonalInfoTCF2PurposeOneTrue(t *testing.T) { - vendorListData := mockVendorListDataTCF2(t, 2, tcf2BasicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -396,8 +416,8 @@ func TestAllowPersonalInfoTCF2PurposeOneTrue(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, @@ -413,6 +433,7 @@ func TestAllowPersonalInfoTCF2PurposeOneTrue(t *testing.T) { consent: "COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA", allowPI: false, allowGeo: false, + allowID: false, }, { description: "Pubmatic vendor test, flex purposes claimed", @@ -420,6 +441,7 @@ func TestAllowPersonalInfoTCF2PurposeOneTrue(t *testing.T) { consent: "COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA", allowPI: true, allowGeo: true, + allowID: true, }, { description: "Rubicon vendor test, Specific purposes/LIs claimed, no geo claimed", @@ -427,19 +449,21 @@ func TestAllowPersonalInfoTCF2PurposeOneTrue(t *testing.T) { consent: "COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA", allowPI: true, allowGeo: false, + allowID: true, }, } for _, td := range testDefs { - allowPI, allowGeo, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) + allowPI, allowGeo, allowID, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) assert.NoErrorf(t, err, "Error processing PersonalInfoAllowed for %s", td.description) assert.EqualValuesf(t, td.allowPI, allowPI, "AllowPI failure on %s", td.description) assert.EqualValuesf(t, td.allowGeo, allowGeo, "AllowGeo failure on %s", td.description) + assert.EqualValuesf(t, td.allowID, allowID, "AllowID failure on %s", td.description) } } func TestAllowPersonalInfoTCF2PurposeOneFalse(t *testing.T) { - vendorListData := mockVendorListDataTCF2(t, 2, tcf2BasicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -448,8 +472,8 @@ func TestAllowPersonalInfoTCF2PurposeOneFalse(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, @@ -458,6 +482,7 @@ func TestAllowPersonalInfoTCF2PurposeOneFalse(t *testing.T) { perms.cfg.TCF2.PurposeOneTreatment.AccessAllowed = false // COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA Purpose one flag set + // Purpose one treatment will fail PI, but allow passing the IDs. testDefs := []tcf2TestDef{ { description: "Appnexus vendor test, insufficient purposes claimed", @@ -465,6 +490,7 @@ func TestAllowPersonalInfoTCF2PurposeOneFalse(t *testing.T) { consent: "COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA", allowPI: false, allowGeo: false, + allowID: false, }, { description: "Pubmatic vendor test, flex purposes claimed", @@ -472,6 +498,7 @@ func TestAllowPersonalInfoTCF2PurposeOneFalse(t *testing.T) { consent: "COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA", allowPI: false, allowGeo: true, + allowID: true, }, { description: "Rubicon vendor test, Specific purposes/LIs claimed, no geo claimed", @@ -479,19 +506,21 @@ func TestAllowPersonalInfoTCF2PurposeOneFalse(t *testing.T) { consent: "COzqiL3OzqiL3NIAAAENAiCMAP_AAH_AAIAAAQEX2S5MAICL7JcmAAA", allowPI: false, allowGeo: false, + allowID: true, }, } for _, td := range testDefs { - allowPI, allowGeo, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) + allowPI, allowGeo, allowID, err := perms.PersonalInfoAllowed(context.Background(), td.bidder, "", td.consent) assert.NoErrorf(t, err, "Error processing PersonalInfoAllowed for %s", td.description) assert.EqualValuesf(t, td.allowPI, allowPI, "AllowPI failure on %s", td.description) assert.EqualValuesf(t, td.allowGeo, allowGeo, "AllowGeo failure on %s", td.description) + assert.EqualValuesf(t, td.allowID, allowID, "AllowID failure on %s", td.description) } } func TestAllowSyncTCF2(t *testing.T) { - vendorListData := mockVendorListDataTCF2(t, 2, tcf2BasicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -500,8 +529,8 @@ func TestAllowSyncTCF2(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, @@ -518,9 +547,9 @@ func TestAllowSyncTCF2(t *testing.T) { } func TestProhibitedPurposeSyncTCF2(t *testing.T) { - basicPurposes := tcf2BasicPurposes - basicPurposes[8] = &purposes{purposes: []int{7}} - vendorListData := mockVendorListDataTCF2(t, 2, basicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + tcf2VendorList34 := buildTCF2VendorList34() + tcf2VendorList34.Vendors["8"].Purposes = []int{7} + vendorListData := tcf2MarshalVendorList(tcf2VendorList34) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -529,15 +558,15 @@ func TestProhibitedPurposeSyncTCF2(t *testing.T) { openrtb_ext.BidderRubicon: 8, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, } perms.cfg.HostVendorID = 8 - // COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA : TCF2 with full consensts to purposes and vendors 2, 6, 8 + // COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA : TCF2 with full consents to purposes for vendors 2, 6, 8 allowSync, err := perms.HostCookiesAllowed(context.Background(), "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA") assert.NoErrorf(t, err, "Error processing HostCookiesAllowed") assert.EqualValuesf(t, false, allowSync, "HostCookiesAllowed failure") @@ -548,9 +577,7 @@ func TestProhibitedPurposeSyncTCF2(t *testing.T) { } func TestProhibitedVendorSyncTCF2(t *testing.T) { - basicPurposes := tcf2BasicPurposes - basicPurposes[10] = &purposes{purposes: []int{1}} - vendorListData := mockVendorListDataTCF2(t, 2, basicPurposes, tcf2LegitInterests, tcf2FlexPurposes, tcf2SpecialPuproses) + vendorListData := tcf2MarshalVendorList(buildTCF2VendorList34()) perms := permissionsImpl{ cfg: tcf2Config, vendorIDs: map[openrtb_ext.BidderName]uint16{ @@ -560,20 +587,21 @@ func TestProhibitedVendorSyncTCF2(t *testing.T) { openrtb_ext.BidderOpenx: 10, }, fetchVendorList: map[uint8]func(ctx context.Context, id uint16) (vendorlist.VendorList, error){ - tCF1: nil, - tCF2: listFetcher(map[uint16]vendorlist.VendorList{ + tcf1SpecVersion: nil, + tcf2SpecVersion: listFetcher(map[uint16]vendorlist.VendorList{ 34: parseVendorListDataV2(t, vendorListData), }), }, } perms.cfg.HostVendorID = 10 - // COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA : TCF2 with full consensts to purposes and vendors 2, 4, 6 + // COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA : TCF2 with full consents to purposes for vendors 2, 6, 8 allowSync, err := perms.HostCookiesAllowed(context.Background(), "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA") assert.NoErrorf(t, err, "Error processing HostCookiesAllowed") assert.EqualValuesf(t, false, allowSync, "HostCookiesAllowed failure") - allowSync, err = perms.BidderSyncAllowed(context.Background(), openrtb_ext.BidderRubicon, "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA") + // Permission disallowed due to consent string not including vendor 10. + allowSync, err = perms.BidderSyncAllowed(context.Background(), openrtb_ext.BidderOpenx, "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA") assert.NoErrorf(t, err, "Error processing BidderSyncAllowed") assert.EqualValuesf(t, false, allowSync, "BidderSyncAllowed failure") } diff --git a/gdpr/vendorlist-fetching.go b/gdpr/vendorlist-fetching.go index 5cbcbfac784..42480041bc1 100644 --- a/gdpr/vendorlist-fetching.go +++ b/gdpr/vendorlist-fetching.go @@ -26,52 +26,83 @@ type saveVendors func(uint16, api.VendorList) // // Nothing in this file is exported. Public APIs can be found in gdpr.go -func newVendorListFetcher(initCtx context.Context, cfg config.GDPR, client *http.Client, urlMaker func(uint16, uint8) string, TCFVer uint8) func(ctx context.Context, id uint16) (vendorlist.VendorList, error) { - // These save and load functions can be used to store & retrieve lists from our cache. - save, load := newVendorListCache() +func newVendorListFetcher(initCtx context.Context, cfg config.GDPR, client *http.Client, urlMaker func(uint16, uint8) string, tcfSpecVersion uint8) func(ctx context.Context, id uint16) (vendorlist.VendorList, error) { + var fallback api.VendorList + if tcfSpecVersion == tcf1SpecVersion && len(cfg.TCF1.FallbackGVLPath) > 0 { + fallback = loadFallbackGVL(cfg.TCF1.FallbackGVLPath) + } - withTimeout, cancel := context.WithTimeout(initCtx, cfg.Timeouts.InitTimeout()) - defer cancel() - populateCache(withTimeout, client, urlMaker, save, TCFVer) + // If we are not going to try fetching the GVL dynamically, we have a simple fetcher. + if !cfg.TCF1.FetchGVL && tcfSpecVersion == tcf1SpecVersion { + if fallback != nil { + return func(ctx context.Context, vendorListVersion uint16) (vendorlist.VendorList, error) { + return fallback, nil + } + } + return func(ctx context.Context, vendorListVersion uint16) (vendorlist.VendorList, error) { + return nil, makeVendorListNotFoundError(vendorListVersion) + } + } + + cacheSave, cacheLoad := newVendorListCache(fallback) - saveOneSometimes := newOccasionalSaver(cfg.Timeouts.ActiveTimeout(), TCFVer) + preloadContext, cancel := context.WithTimeout(initCtx, cfg.Timeouts.InitTimeout()) + defer cancel() + preloadCache(preloadContext, client, urlMaker, cacheSave, tcfSpecVersion) - return func(ctx context.Context, id uint16) (vendorlist.VendorList, error) { - list := load(id) - if list != nil { + saveOneRateLimited := newOccasionalSaver(cfg.Timeouts.ActiveTimeout(), tcfSpecVersion) + return func(ctx context.Context, vendorListVersion uint16) (vendorlist.VendorList, error) { + // Attempt To Load From Cache + if list := cacheLoad(vendorListVersion); list != nil { return list, nil } - saveOneSometimes(ctx, client, urlMaker(id, TCFVer), save) - list = load(id) - if list != nil { + + // Attempt To Download + // - May not add to cache immediately. + saveOneRateLimited(ctx, client, urlMaker(vendorListVersion, tcfSpecVersion), cacheSave) + + // Attempt To Load From Cache Again + // - May have been added by the call to saveOneRateLimited. + if list := cacheLoad(vendorListVersion); list != nil { return list, nil } - return nil, fmt.Errorf("gdpr vendor list version %d does not exist, or has not been loaded yet. Try again in a few minutes", id) + + // Attempt To Use Hardcoded Fallback + if fallback != nil { + return fallback, nil + } + + // Give Up + return nil, makeVendorListNotFoundError(vendorListVersion) } } -// populateCache saves all the known versions of the vendor list for future use. -func populateCache(ctx context.Context, client *http.Client, urlMaker func(uint16, uint8) string, saver saveVendors, TCFVer uint8) { - latestVersion := saveOne(ctx, client, urlMaker(0, TCFVer), saver, TCFVer) +func makeVendorListNotFoundError(vendorListVersion uint16) error { + return fmt.Errorf("gdpr vendor list version %d does not exist, or has not been loaded yet. Try again in a few minutes", vendorListVersion) +} + +// preloadCache saves all the known versions of the vendor list for future use. +func preloadCache(ctx context.Context, client *http.Client, urlMaker func(uint16, uint8) string, saver saveVendors, tcfSpecVersion uint8) { + latestVersion := saveOne(ctx, client, urlMaker(0, tcfSpecVersion), saver, tcfSpecVersion) for i := uint16(1); i < latestVersion; i++ { - saveOne(ctx, client, urlMaker(i, TCFVer), saver, TCFVer) + saveOne(ctx, client, urlMaker(i, tcfSpecVersion), saver, tcfSpecVersion) } } // Make a URL which can be used to fetch a given version of the Global Vendor List. If the version is 0, // this will fetch the latest version. -func vendorListURLMaker(version uint16, TCFVer uint8) string { - if TCFVer == 2 { - if version == 0 { +func vendorListURLMaker(vendorListVersion uint16, tcfSpecVersion uint8) string { + if tcfSpecVersion == tcf2SpecVersion { + if vendorListVersion == 0 { return "https://vendorlist.consensu.org/v2/vendor-list.json" } - return "https://vendorlist.consensu.org/v2/archives/vendor-list-v" + strconv.Itoa(int(version)) + ".json" + return "https://vendorlist.consensu.org/v2/archives/vendor-list-v" + strconv.Itoa(int(vendorListVersion)) + ".json" } - if version == 0 { + if vendorListVersion == 0 { return "https://vendorlist.consensu.org/vendorlist.json" } - return "https://vendorlist.consensu.org/v-" + strconv.Itoa(int(version)) + "/vendorlist.json" + return "https://vendorlist.consensu.org/v-" + strconv.Itoa(int(vendorListVersion)) + "/vendorlist.json" } // newOccasionalSaver returns a wrapped version of saveOne() which only activates every few minutes. @@ -79,22 +110,24 @@ func vendorListURLMaker(version uint16, TCFVer uint8) string { // The goal here is to update quickly when new versions of the VendorList are released, but not wreck // server performance if a bad CMP starts sending us malformed consent strings that advertize a version // that doesn't exist yet. -func newOccasionalSaver(timeout time.Duration, TCFVer uint8) func(ctx context.Context, client *http.Client, url string, saver saveVendors) { +func newOccasionalSaver(timeout time.Duration, tcfSpecVersion uint8) func(ctx context.Context, client *http.Client, url string, saver saveVendors) { lastSaved := &atomic.Value{} lastSaved.Store(time.Time{}) return func(ctx context.Context, client *http.Client, url string, saver saveVendors) { now := time.Now() - if now.Sub(lastSaved.Load().(time.Time)).Minutes() > 10 { + timeSinceLastSave := now.Sub(lastSaved.Load().(time.Time)) + + if timeSinceLastSave.Minutes() > 10 { withTimeout, cancel := context.WithTimeout(ctx, timeout) defer cancel() - saveOne(withTimeout, client, url, saver, TCFVer) + saveOne(withTimeout, client, url, saver, tcfSpecVersion) lastSaved.Store(now) } } } -func saveOne(ctx context.Context, client *http.Client, url string, saver saveVendors, cTFVer uint8) uint16 { +func saveOne(ctx context.Context, client *http.Client, url string, saver saveVendors, tcfSpecVersion uint8) uint16 { req, err := http.NewRequest("GET", url, nil) if err != nil { glog.Errorf("Failed to build GET %s request. Cookie syncs may be affected: %v", url, err) @@ -118,7 +151,7 @@ func saveOne(ctx context.Context, client *http.Client, url string, saver saveVen return 0 } var newList api.VendorList - if cTFVer == 2 { + if tcfSpecVersion == tcf2SpecVersion { newList, err = vendorlist2.ParseEagerly(respBody) } else { newList, err = vendorlist.ParseEagerly(respBody) @@ -132,14 +165,15 @@ func saveOne(ctx context.Context, client *http.Client, url string, saver saveVen return newList.Version() } -func newVendorListCache() (save func(id uint16, list api.VendorList), load func(id uint16) api.VendorList) { +func newVendorListCache(fallbackVL api.VendorList) (save func(vendorListVersion uint16, list api.VendorList), load func(vendorListVersion uint16) api.VendorList) { cache := &sync.Map{} - save = func(id uint16, list api.VendorList) { - cache.Store(id, list) + save = func(vendorListVersion uint16, list api.VendorList) { + cache.Store(vendorListVersion, list) } - load = func(id uint16) api.VendorList { - list, ok := cache.Load(id) + + load = func(vendorListVersion uint16) api.VendorList { + list, ok := cache.Load(vendorListVersion) if ok { return list.(vendorlist.VendorList) } @@ -147,3 +181,16 @@ func newVendorListCache() (save func(id uint16, list api.VendorList), load func( } return } + +func loadFallbackGVL(fallbackGVLPath string) vendorlist.VendorList { + fallbackContents, err := ioutil.ReadFile(fallbackGVLPath) + if err != nil { + glog.Fatalf("Error reading from file %s: %v", fallbackGVLPath, err) + } + + fallback, err := vendorlist.ParseEagerly(fallbackContents) + if err != nil { + glog.Fatalf("Error processing default GVL from %s: %v", fallbackGVLPath, err) + } + return fallback +} diff --git a/gdpr/vendorlist-fetching_test.go b/gdpr/vendorlist-fetching_test.go index 32d7ef351b3..6329d8fb69c 100644 --- a/gdpr/vendorlist-fetching_test.go +++ b/gdpr/vendorlist-fetching_test.go @@ -7,136 +7,697 @@ import ( "net/http/httptest" "strconv" "testing" - "time" + + "github.com/stretchr/testify/assert" "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/prebid/go-gdpr/consentconstants" ) -func TestVendorFetch(t *testing.T) { - vendorListOne := mockVendorListData(t, 1, map[uint16]*purposes{ - 32: { - purposes: []int{1, 2}, +func TestTCF1FetcherInitialLoad(t *testing.T) { + // Loads two vendor lists during initialization by setting the latest vendor list version to 2. + + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 2, + vendorLists: map[int]string{ + 1: tcf1VendorList1, + 2: tcf1VendorList2, + }, + }))) + defer server.Close() + + testCases := []test{ + { + description: "Fetch - No Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 1, + }, + expected: vendorList1Expected, + }, + { + description: "Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "Fetch - Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 1, + }, + expected: vendorList1Expected, + }, + { + description: "Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "No Fetch - Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 1, + }, + expected: vendorListFallbackExpected, + }, + { + description: "No Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorListFallbackExpected, }, - }) - vendorListTwo := mockVendorListData(t, 2, map[uint16]*purposes{ - 32: { - purposes: []int{1, 2, 3}, + { + description: "No Fetch - No Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 1, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 1 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + { + description: "No Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + } + + for _, test := range testCases { + runTest(t, test, tcf1SpecVersion, server) + } +} + +func TestTCF2FetcherInitialLoad(t *testing.T) { + // Loads two vendor lists during initialization by setting the latest vendor list version to 2. + // Ensures TCF1 fetch settings have no effect on TCF2. + + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 2, + vendorLists: map[int]string{ + 1: tcf2VendorList1, + 2: tcf2VendorList2, }, - }) - server := httptest.NewServer(http.HandlerFunc(mockServer(2, map[int]string{ - 1: vendorListOne, - 2: vendorListTwo, }))) defer server.Close() - fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), 1) - list, err := fetcher(context.Background(), 1) - assertNilErr(t, err) - vendor := list.Vendor(32) - assertBoolsEqual(t, true, vendor.Purpose(1)) - assertBoolsEqual(t, false, vendor.Purpose(3)) - assertBoolsEqual(t, false, vendor.Purpose(4)) - - list, err = fetcher(context.Background(), 2) - assertNilErr(t, err) - vendor = list.Vendor(32) - assertBoolsEqual(t, true, vendor.Purpose(1)) - assertBoolsEqual(t, true, vendor.Purpose(3)) -} - -func TestLazyFetch(t *testing.T) { - firstVendorList := mockVendorListData(t, 1, map[uint16]*purposes{ - 32: { - purposes: []int{1, 2}, - }, - }) - secondVendorList := mockVendorListData(t, 2, map[uint16]*purposes{ - 3: { - purposes: []int{1}, - }, - }) - server := httptest.NewServer(http.HandlerFunc(mockServer(1, map[int]string{ - 1: firstVendorList, - 2: secondVendorList, + testCases := []test{ + { + description: "Fetch - No Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 1, + }, + expected: vendorList1Expected, + }, + { + description: "Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "Fetch - Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 1, + }, + expected: vendorList1Expected, + }, + { + description: "Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "No Fetch - Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 1, + }, + expected: vendorList1Expected, + }, + { + description: "No Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "No Fetch - No Fallback - Vendor List 1", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 1, + }, + expected: vendorList1Expected, + }, + { + description: "No Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + } + + for _, test := range testCases { + runTest(t, test, tcf2SpecVersion, server) + } +} + +func TestTCF1FetcherDynamicLoadListExists(t *testing.T) { + // Loads the first vendor list during initialization by setting the latest vendor list version to 1. + // All other vendor lists will be dynamically loaded. + + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: tcf1VendorList1, + 2: tcf1VendorList2, + }, }))) defer server.Close() - fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), 1) - list, err := fetcher(context.Background(), 2) - assertNilErr(t, err) + testCases := []test{ + { + description: "Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "No Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorListFallbackExpected, + }, + { + description: "No Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + } - vendor := list.Vendor(3) - assertBoolsEqual(t, true, vendor.Purpose(1)) - assertBoolsEqual(t, false, vendor.Purpose(2)) + for _, test := range testCases { + runTest(t, test, tcf1SpecVersion, server) + } } -func TestInitialTimeout(t *testing.T) { - list := mockVendorListData(t, 1, map[uint16]*purposes{ - 32: { - purposes: []int{1, 2}, +func TestTCF2FetcherDynamicLoadListExists(t *testing.T) { + // Loads the first vendor list during initialization by setting the latest vendor list version to 1. + // All other vendor lists will be dynamically loaded. + // Ensures TCF1 fetch settings have no effect on TCF2. + + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: tcf2VendorList1, + 2: tcf2VendorList2, }, - }) - server := httptest.NewServer(http.HandlerFunc(mockServer(1, map[int]string{ - 1: list, }))) defer server.Close() - ctx, cancel := context.WithDeadline(context.Background(), time.Time{}) - defer cancel() - fetcher := newVendorListFetcher(ctx, testConfig(), server.Client(), testURLMaker(server), 1) - _, err := fetcher(context.Background(), 1) // This should do a lazy fetch, even though the initial call failed - assertNilErr(t, err) + testCases := []test{ + { + description: "Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "No Fetch - Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + { + description: "No Fetch - No Fallback - Vendor List 2", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: vendorList2Expected, + }, + } + + for _, test := range testCases { + runTest(t, test, tcf2SpecVersion, server) + } } -func TestFetchThrottling(t *testing.T) { - vendorListTwo := mockVendorListData(t, 2, map[uint16]*purposes{ - 32: { - purposes: []int{1, 2}, +func TestTCF1FetcherDynamicLoadListDoesntExist(t *testing.T) { + // Loads the first vendor list during initialization by setting the latest vendor list version to 1. + // All other vendor list load attempts will be done dynamically. + + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: tcf1VendorList1, + }, + }))) + defer server.Close() + + testCases := []test{ + { + description: "Fetch - No Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + { + description: "Fetch - Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorListFallbackExpected, + }, + { + description: "No Fetch - Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: vendorListFallbackExpected, }, - }) - vendorListThree := mockVendorListData(t, 3, map[uint16]*purposes{ - 32: { - purposes: []int{1, 2}, + { + description: "No Fetch - No Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + } + + for _, test := range testCases { + runTest(t, test, 1, server) + } +} + +func TestTCF2FetcherDynamicLoadListDoesntExist(t *testing.T) { + // Loads the first vendor list during initialization by setting the latest vendor list version to 1. + // All other vendor list load attempts will be done dynamically. + // Ensures TCF1 fetch settings have no effect on TCF2. + + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: tcf2VendorList1, }, - }) - server := httptest.NewServer(http.HandlerFunc(mockServer(1, map[int]string{ - 1: "{}", - 2: vendorListTwo, - 3: vendorListThree, }))) defer server.Close() - fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), 1) - _, err := fetcher(context.Background(), 2) - assertNilErr(t, err) - _, err = fetcher(context.Background(), 3) - assertErr(t, err, false) + testCases := []test{ + { + description: "Fetch - No Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + { + description: "Fetch - Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: true, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + { + description: "No Fetch - Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: true, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + { + description: "No Fetch - No Fallback - Vendor Doesn't Exist", + setup: testSetup{ + enableTCF1Fetch: false, + enableTCF1Fallback: false, + vendorListVersion: 2, + }, + expected: testExpected{ + errorMessage: "gdpr vendor list version 2 does not exist, or has not been loaded yet. Try again in a few minutes", + }, + }, + } + + for _, test := range testCases { + runTest(t, test, tcf2SpecVersion, server) + } +} + +func TestTCF1FetcherThrottling(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 1, + Vendors: []tcf1Vendor{{ID: 12, Purposes: []int{1}}}, + }), + 2: tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 2, + Vendors: []tcf1Vendor{{ID: 12, Purposes: []int{1, 2}}}, + }), + 3: tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 3, + Vendors: []tcf1Vendor{{ID: 12, Purposes: []int{1, 2, 3}}}, + }), + }, + }))) + defer server.Close() + + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), tcf1SpecVersion) + + // Dynamically Load List 2 Successfully + _, errList1 := fetcher(context.Background(), 2) + assert.NoError(t, errList1) + + // Fail To Load List 3 Due To Rate Limiting + // - The request is rate limited after dynamically list 2. + _, errList2 := fetcher(context.Background(), 3) + assert.EqualError(t, errList2, "gdpr vendor list version 3 does not exist, or has not been loaded yet. Try again in a few minutes") } -func TestMalformedVendorlistFetch(t *testing.T) { - server := httptest.NewServer(http.HandlerFunc(mockServer(1, map[int]string{1: "{}"}))) +func TestTCF2FetcherThrottling(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: tcf2MarshalVendorList(tcf2VendorList{ + VendorListVersion: 1, + Vendors: map[string]*tcf2Vendor{"12": {ID: 12, Purposes: []int{1}}}, + }), + 2: tcf2MarshalVendorList(tcf2VendorList{ + VendorListVersion: 2, + Vendors: map[string]*tcf2Vendor{"12": {ID: 12, Purposes: []int{1, 2}}}, + }), + 3: tcf2MarshalVendorList(tcf2VendorList{ + VendorListVersion: 3, + Vendors: map[string]*tcf2Vendor{"12": {ID: 12, Purposes: []int{1, 2, 3}}}, + }), + }, + }))) + defer server.Close() + + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), tcf2SpecVersion) + + // Dynamically Load List 2 Successfully + _, errList1 := fetcher(context.Background(), 2) + assert.NoError(t, errList1) + + // Fail To Load List 3 Due To Rate Limiting + // - The request is rate limited after dynamically list 2. + _, errList2 := fetcher(context.Background(), 3) + assert.EqualError(t, errList2, "gdpr vendor list version 3 does not exist, or has not been loaded yet. Try again in a few minutes") +} + +func TestTCF1MalformedVendorlist(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: "malformed", + }, + }))) defer server.Close() - fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), 1) + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), tcf1SpecVersion) _, err := fetcher(context.Background(), 1) - assertErr(t, err, false) + + // Fetching should fail since vendor list could not be unmarshalled. + assert.Error(t, err) } -func TestMissingVendorlistFetch(t *testing.T) { - server := httptest.NewServer(http.HandlerFunc(mockServer(1, map[int]string{1: "{}"}))) +func TestTCF2MalformedVendorlist(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(mockServer(serverSettings{ + vendorListLatestVersion: 1, + vendorLists: map[int]string{ + 1: "malformed", + }, + }))) defer server.Close() - fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), 1) - _, err := fetcher(context.Background(), 2) - assertErr(t, err, false) + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), tcf2SpecVersion) + _, err := fetcher(context.Background(), 1) + + // Fetching should fail since vendor list could not be unmarshalled. + assert.Error(t, err) +} + +func TestTCF1ServerUrlInvalid(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})) + server.Close() + + invalidURLGenerator := func(uint16, uint8) string { return " http://invalid-url-has-leading-whitespace" } + + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), invalidURLGenerator, tcf1SpecVersion) + _, err := fetcher(context.Background(), 1) + + assert.EqualError(t, err, "gdpr vendor list version 1 does not exist, or has not been loaded yet. Try again in a few minutes") +} + +func TestTCF2ServerUrlInvalid(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})) + server.Close() + + invalidURLGenerator := func(uint16, uint8) string { return " http://invalid-url-has-leading-whitespace" } + + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), invalidURLGenerator, tcf2SpecVersion) + _, err := fetcher(context.Background(), 1) + + assert.EqualError(t, err, "gdpr vendor list version 1 does not exist, or has not been loaded yet. Try again in a few minutes") +} + +func TestTCF1ServerUnavailable(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})) + server.Close() + + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), tcf1SpecVersion) + _, err := fetcher(context.Background(), 1) + + assert.EqualError(t, err, "gdpr vendor list version 1 does not exist, or has not been loaded yet. Try again in a few minutes") +} + +func TestTCF2ServerUnavailable(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})) + server.Close() + + fetcher := newVendorListFetcher(context.Background(), testConfig(), server.Client(), testURLMaker(server), tcf2SpecVersion) + _, err := fetcher(context.Background(), 1) + + assert.EqualError(t, err, "gdpr vendor list version 1 does not exist, or has not been loaded yet. Try again in a few minutes") +} + +func TestVendorListURLMaker(t *testing.T) { + testCases := []struct { + description string + tcfSpecVersion uint8 + vendorListVersion uint16 + expectedURL string + }{ + { + description: "TCF1 - Latest", + tcfSpecVersion: 1, + vendorListVersion: 0, // Forces latest version. + expectedURL: "https://vendorlist.consensu.org/vendorlist.json", + }, + { + description: "TCF1 - Specific", + tcfSpecVersion: 1, + vendorListVersion: 42, + expectedURL: "https://vendorlist.consensu.org/v-42/vendorlist.json", + }, + { + description: "TCF2 - Latest", + tcfSpecVersion: 2, + vendorListVersion: 0, // Forces latest version. + expectedURL: "https://vendorlist.consensu.org/v2/vendor-list.json", + }, + { + description: "TCF2 - Specific", + tcfSpecVersion: 2, + vendorListVersion: 42, + expectedURL: "https://vendorlist.consensu.org/v2/archives/vendor-list-v42.json", + }, + } + + for _, test := range testCases { + result := vendorListURLMaker(test.vendorListVersion, test.tcfSpecVersion) + assert.Equal(t, test.expectedURL, result) + } +} + +var tcf1VendorList1 = tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 1, + Vendors: []tcf1Vendor{{ID: 12, Purposes: []int{2}}}, +}) + +var tcf2VendorList1 = tcf2MarshalVendorList(tcf2VendorList{ + VendorListVersion: 1, + Vendors: map[string]*tcf2Vendor{"12": {ID: 12, Purposes: []int{2}}}, +}) + +var vendorList1Expected = testExpected{ + vendorListVersion: 1, + vendorID: 12, + vendorPurposes: map[int]bool{1: false, 2: true, 3: false}, +} + +var tcf1VendorList2 = tcf1MarshalVendorList(tcf1VendorList{ + VendorListVersion: 2, + Vendors: []tcf1Vendor{{ID: 12, Purposes: []int{2, 3}}}, +}) + +var tcf2VendorList2 = tcf2MarshalVendorList(tcf2VendorList{ + VendorListVersion: 2, + Vendors: map[string]*tcf2Vendor{"12": {ID: 12, Purposes: []int{2, 3}}}, +}) + +var vendorList2Expected = testExpected{ + vendorListVersion: 2, + vendorID: 12, + vendorPurposes: map[int]bool{1: false, 2: true, 3: true}, } -func TestVendorListMaker(t *testing.T) { - assertStringsEqual(t, "https://vendorlist.consensu.org/vendorlist.json", vendorListURLMaker(0, 1)) - assertStringsEqual(t, "https://vendorlist.consensu.org/v-2/vendorlist.json", vendorListURLMaker(2, 1)) - assertStringsEqual(t, "https://vendorlist.consensu.org/v-12/vendorlist.json", vendorListURLMaker(12, 1)) - assertStringsEqual(t, "https://vendorlist.consensu.org/v2/vendor-list.json", vendorListURLMaker(0, 2)) - assertStringsEqual(t, "https://vendorlist.consensu.org/v2/archives/vendor-list-v7.json", vendorListURLMaker(7, 2)) +var vendorListFallbackExpected = testExpected{ + vendorListVersion: 215, // Values from hardcoded fallback file. + vendorID: 12, + vendorPurposes: map[int]bool{1: true, 2: false, 3: true}, +} + +type tcf1VendorList struct { + VendorListVersion uint16 `json:"vendorListVersion"` + Vendors []tcf1Vendor `json:"vendors"` +} + +type tcf1Vendor struct { + ID uint16 `json:"id"` + Purposes []int `json:"purposeIds"` +} + +func tcf1MarshalVendorList(vendorList tcf1VendorList) string { + json, _ := json.Marshal(vendorList) + return string(json) +} + +type tcf2VendorList struct { + VendorListVersion uint16 `json:"vendorListVersion"` + Vendors map[string]*tcf2Vendor `json:"vendors"` +} + +type tcf2Vendor struct { + ID uint16 `json:"id"` + Purposes []int `json:"purposes"` + LegIntPurposes []int `json:"legIntPurposes"` + FlexiblePurposes []int `json:"flexiblePurposes"` + SpecialPurposes []int `json:"specialPurposes"` +} + +func tcf2MarshalVendorList(vendorList tcf2VendorList) string { + json, _ := json.Marshal(vendorList) + return string(json) +} + +type serverSettings struct { + vendorListLatestVersion int + vendorLists map[int]string } // mockServer returns a handler which returns the given response for each global vendor list version. @@ -150,129 +711,74 @@ func TestVendorListMaker(t *testing.T) { // // If the "version" query param points to a version which doesn't exist, it returns a 403. // Don't ask why... that's just what the official page is doing. See https://vendorlist.consensu.org/v-9999/vendorlist.json -func mockServer(latestVersion int, responses map[int]string) func(http.ResponseWriter, *http.Request) { +func mockServer(settings serverSettings) func(http.ResponseWriter, *http.Request) { return func(w http.ResponseWriter, req *http.Request) { - version := req.URL.Query().Get("version") - versionInt, err := strconv.Atoi(version) + vendorListVersion := req.URL.Query().Get("version") + vendorListVersionInt, err := strconv.Atoi(vendorListVersion) if err != nil { w.WriteHeader(http.StatusBadRequest) - w.Write([]byte("Request had invalid version: " + version)) + w.Write([]byte("Request had invalid version: " + vendorListVersion)) return } - if versionInt == 0 { - versionInt = latestVersion + if vendorListVersionInt == 0 { + vendorListVersionInt = settings.vendorListLatestVersion } - response, ok := responses[versionInt] + response, ok := settings.vendorLists[vendorListVersionInt] if !ok { w.WriteHeader(http.StatusForbidden) - w.Write([]byte("Version not found: " + version)) + w.Write([]byte("Version not found: " + vendorListVersion)) return } w.Write([]byte(response)) } } -func mockVendorListData(t *testing.T, version uint16, vendors map[uint16]*purposes) string { - type vendorContract struct { - ID uint16 `json:"id"` - Purposes []int `json:"purposeIds"` - } - - type vendorListContract struct { - Version uint16 `json:"vendorListVersion"` - Vendors []vendorContract `json:"vendors"` - } - - buildVendors := func(input map[uint16]*purposes) []vendorContract { - vendors := make([]vendorContract, 0, len(input)) - for id, purpose := range input { - vendors = append(vendors, vendorContract{ - ID: id, - Purposes: purpose.purposes, - }) - } - return vendors - } - - obj := vendorListContract{ - Version: version, - Vendors: buildVendors(vendors), - } - data, err := json.Marshal(obj) - assertNilErr(t, err) - return string(data) +type test struct { + description string + setup testSetup + expected testExpected } -type purposeMap map[uint16]*purposes - -func mockVendorListDataTCF2(t *testing.T, version uint16, basicPurposes purposeMap, legitInterests purposeMap, flexPurposes purposeMap, specialPurposes purposeMap) string { - type vendorContract struct { - ID uint16 `json:"id"` - Purposes []int `json:"purposes"` - LegIntPurposes []int `json:"legIntPurposes"` - FlexiblePurposes []int `json:"flexiblePurposes"` - SpecialPurposes []int `json:"specialPurposes"` - } - - type vendorListContract struct { - Version uint16 `json:"vendorListVersion"` - Vendors map[string]vendorContract `json:"vendors"` - } - - vendors := make(map[string]vendorContract, len(basicPurposes)) - for id, purpose := range basicPurposes { - sid := strconv.Itoa(int(id)) - vendor, ok := vendors[sid] - if !ok { - vendor = vendorContract{ID: id} - } - vendor.Purposes = purpose.purposes - vendors[sid] = vendor - } +type testSetup struct { + enableTCF1Fetch bool + enableTCF1Fallback bool + vendorListVersion uint16 +} - for id, purpose := range legitInterests { - sid := strconv.Itoa(int(id)) - vendor, ok := vendors[sid] - if !ok { - vendor = vendorContract{ID: id} - } - vendor.LegIntPurposes = purpose.purposes - vendors[sid] = vendor - } +type testExpected struct { + errorMessage string + vendorListVersion uint16 + vendorID uint16 + vendorPurposes map[int]bool +} - for id, purpose := range flexPurposes { - sid := strconv.Itoa(int(id)) - vendor, ok := vendors[sid] - if !ok { - vendor = vendorContract{ID: id} - } - vendor.FlexiblePurposes = purpose.purposes - vendors[sid] = vendor +func runTest(t *testing.T, test test, tcfSpecVersion uint8, server *httptest.Server) { + config := testConfig() + config.TCF1.FetchGVL = test.setup.enableTCF1Fetch + if test.setup.enableTCF1Fallback { + config.TCF1.FallbackGVLPath = "../static/tcf1/fallback_gvl.json" } - for id, purpose := range specialPurposes { - sid := strconv.Itoa(int(id)) - vendor, ok := vendors[sid] - if !ok { - vendor = vendorContract{ID: id} + fetcher := newVendorListFetcher(context.Background(), config, server.Client(), testURLMaker(server), tcfSpecVersion) + vendorList, err := fetcher(context.Background(), test.setup.vendorListVersion) + + if test.expected.errorMessage != "" { + assert.EqualError(t, err, test.expected.errorMessage, test.description+":error") + } else { + assert.NoError(t, err, test.description+":vendorlist") + assert.Equal(t, test.expected.vendorListVersion, vendorList.Version(), test.description+":vendorlistid") + vendor := vendorList.Vendor(test.expected.vendorID) + for id, expected := range test.expected.vendorPurposes { + result := vendor.Purpose(consentconstants.Purpose(id)) + assert.Equalf(t, expected, result, "%s:vendor-%d:purpose-%d", test.description, vendorList.Version(), id) } - vendor.SpecialPurposes = purpose.purposes - vendors[sid] = vendor } - - obj := vendorListContract{ - Version: version, - Vendors: vendors, - } - data, err := json.Marshal(obj) - assertNilErr(t, err) - return string(data) } func testURLMaker(server *httptest.Server) func(uint16, uint8) string { url := server.URL - return func(version uint16, TCFVer uint8) string { - return url + "?version=" + strconv.Itoa(int(version)) + return func(vendorListVersion uint16, tcfSpecVersion uint8) string { + return url + "?version=" + strconv.Itoa(int(vendorListVersion)) } } @@ -282,9 +788,8 @@ func testConfig() config.GDPR { InitVendorlistFetch: 60 * 1000, ActiveVendorlistFetch: 1000 * 5, }, + TCF1: config.TCF1{ + FetchGVL: true, + }, } } - -type purposes struct { - purposes []int -} diff --git a/go.mod b/go.mod index 949125b8594..41bd4a9384f 100644 --- a/go.mod +++ b/go.mod @@ -16,6 +16,7 @@ require ( github.com/cespare/xxhash v1.0.0 // indirect github.com/chasex/glog v0.0.0-20160217080310-c62392af379c github.com/coocood/freecache v1.0.1 + github.com/docker/go-units v0.4.0 github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 github.com/evanphx/json-patch v0.0.0-20180720181644-f195058310bd github.com/gofrs/uuid v3.2.0+incompatible @@ -33,7 +34,7 @@ require ( github.com/onsi/ginkgo v1.10.1 // indirect github.com/onsi/gomega v1.7.0 // indirect github.com/pelletier/go-toml v1.2.0 // indirect - github.com/prebid/go-gdpr v0.8.2 + github.com/prebid/go-gdpr v0.8.3 github.com/prometheus/client_golang v0.0.0-20180623155954-77e8f2ddcfed github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e // indirect diff --git a/go.sum b/go.sum index 98713ba6857..37a7df16073 100644 --- a/go.sum +++ b/go.sum @@ -16,6 +16,7 @@ github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLM github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= +github.com/buger/jsonparser v0.0.0-20180318095312-2cac668e8456/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s= github.com/buger/jsonparser v0.0.0-20180808090653-f4dd9f5a6b44 h1:y853v6rXx+zefEcjET3JuKAqvhj+FKflQijjeaSv2iA= github.com/buger/jsonparser v0.0.0-20180808090653-f4dd9f5a6b44/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s= github.com/cespare/xxhash v1.0.0 h1:naDmySfoNg0nKS62/ujM6e71ZgM2AoVdaqGwMG0w18A= @@ -27,6 +28,8 @@ github.com/coocood/freecache v1.0.1/go.mod h1:ePwxCDzOYvARfHdr1pByNct1at3CoKnsip github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw= +github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 h1:Yzb9+7DPaBjB8zlTR87/ElzFsnQfuHnVUVqpZZIcV5Y= github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5/go.mod h1:a2zkGnVExMxdzMo3M0Hi/3sEU+cWnZpSni0O6/Yb/P0= github.com/evanphx/json-patch v0.0.0-20180720181644-f195058310bd h1:biTJQdqouE5by89AAffXG8++TY+9Fsdrg5rinbt3tHk= @@ -72,8 +75,8 @@ github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prebid/go-gdpr v0.8.2 h1:mN2jKYZZpJkCYFQB/nDTJoPpuGYblOYP2UUzOzRggII= -github.com/prebid/go-gdpr v0.8.2/go.mod h1:FPY0uxSrl9/Mz237LnPo3ge4aCG0wQ9FWf2b4WhwNn0= +github.com/prebid/go-gdpr v0.8.3 h1:rjCZNV0AdKygiGHpVhNB42usjEpTN3qidXUPB1yarb0= +github.com/prebid/go-gdpr v0.8.3/go.mod h1:TGzgqQDGKOVUkbqmY25K4uvcwMywSddXEaY4zUFiVBQ= github.com/prometheus/client_golang v0.0.0-20180623155954-77e8f2ddcfed h1:0dloFFFNNDG7c+8qtkYw2FdADrWy9s5cI8wHp6tK3Mg= github.com/prometheus/client_golang v0.0.0-20180623155954-77e8f2ddcfed/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 h1:idejC8f05m9MGOsuEi1ATq9shN03HrxNkD/luQvxCv8= diff --git a/macros/macros.go b/macros/macros.go index a9f77ea95fa..5d6bd7af65e 100644 --- a/macros/macros.go +++ b/macros/macros.go @@ -12,6 +12,7 @@ type EndpointTemplateParams struct { ZoneID string SourceId string AccountID string + AdUnit string } // UserSyncTemplateParams specifies params for an user sync URL template diff --git a/main.go b/main.go index 802714590e0..7312fc99c98 100644 --- a/main.go +++ b/main.go @@ -11,7 +11,7 @@ import ( pbc "github.com/PubMatic-OpenWrap/prebid-server/prebid_cache_client" "github.com/PubMatic-OpenWrap/prebid-server/router" "github.com/PubMatic-OpenWrap/prebid-server/usersync" - "github.com/julienschmidt/httprouter" + "github.com/PubMatic-OpenWrap/prebid-server/util/task" "github.com/golang/glog" "github.com/spf13/viper" @@ -75,7 +75,11 @@ func loadConfig(configFileName string) (*config.Configuration, error) { func serve(revision string, cfg *config.Configuration) error { fetchingInterval := time.Duration(cfg.CurrencyConverter.FetchIntervalSeconds) * time.Second - currencyConverter := currencies.NewRateConverter(&http.Client{}, cfg.CurrencyConverter.FetchURL, fetchingInterval) + staleRatesThreshold := time.Duration(cfg.CurrencyConverter.StaleRatesSeconds) * time.Second + currencyConverter := currencies.NewRateConverter(&http.Client{}, cfg.CurrencyConverter.FetchURL, staleRatesThreshold) + + currencyConverterTickerTask := task.NewTickerTask(fetchingInterval, currencyConverter) + currencyConverterTickerTask.Start() _, err := router.New(cfg, currencyConverter) if err != nil { @@ -85,9 +89,9 @@ func serve(revision string, cfg *config.Configuration) error { pbc.InitPrebidCache(cfg.CacheURL.GetBaseURL()) pbc.InitPrebidCacheURL(cfg.ExternalURL) - // Add cors support //corsRouter := router.SupportCORS(r) - //server.Listen(cfg, router.NoCache{Handler: corsRouter}, router.Admin(revision, currencyConverter), r.MetricsEngine) + //server.Listen(cfg, router.NoCache{Handler: corsRouter}, router.Admin(revision, currencyConverter, fetchingInterval), r.MetricsEngine) + //r.Shutdown() return nil } @@ -112,7 +116,7 @@ func SetUIDS(w http.ResponseWriter, r *http.Request) { router.SetUIDSWrapper(w, r) } -func CookieSync(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { +func CookieSync(w http.ResponseWriter, r *http.Request) { router.CookieSync(w, r) } diff --git a/main_test.go b/main_test.go index 7888d85062f..f3b6748ba48 100644 --- a/main_test.go +++ b/main_test.go @@ -5,6 +5,7 @@ import ( "testing" "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/stretchr/testify/assert" "github.com/spf13/viper" ) @@ -56,10 +57,11 @@ func TestViperEnv(t *testing.T) { ttl := forceEnv(t, "PBS_HOST_COOKIE_TTL_DAYS", "60") defer ttl() - // Basic config set - compareStrings(t, "Viper error: port expected to be %s, found %s", "7777", v.Get("port").(string)) - // Nested config set - compareStrings(t, "Viper error: adapters.pubmatic.endpoint expected to be %s, found %s", "not_an_endpoint", v.Get("adapters.pubmatic.endpoint").(string)) - // Config set with underscores - compareStrings(t, "Viper error: host_cookie.ttl_days expected to be %s, found %s", "60", v.Get("host_cookie.ttl_days").(string)) + ipv4Networks := forceEnv(t, "PBS_REQUEST_VALIDATION_IPV4_PRIVATE_NETWORKS", "1.1.1.1/24 2.2.2.2/24") + defer ipv4Networks() + + assert.Equal(t, 7777, v.Get("port"), "Basic Config") + assert.Equal(t, "not_an_endpoint", v.Get("adapters.pubmatic.endpoint"), "Nested Config") + assert.Equal(t, 60, v.Get("host_cookie.ttl_days"), "Config With Underscores") + assert.ElementsMatch(t, []string{"1.1.1.1/24", "2.2.2.2/24"}, v.Get("request_validation.ipv4_private_networks"), "Arrays") } diff --git a/openrtb_ext/bid.go b/openrtb_ext/bid.go index c1ba5bdbadb..75d83c9d7dd 100644 --- a/openrtb_ext/bid.go +++ b/openrtb_ext/bid.go @@ -13,6 +13,7 @@ type ExtBid struct { // ExtBidPrebid defines the contract for bidresponse.seatbid.bid[i].ext.prebid // DealPriority represents priority of deal bid. If its non deal bid then value will be 0 +// DealTierSatisfied true represents corresponding bid has satisfied the deal tier type ExtBidPrebid struct { Cache *ExtBidPrebidCache `json:"cache,omitempty"` Targeting map[string]string `json:"targeting,omitempty"` @@ -102,6 +103,9 @@ const ( HbSizeConstantKey TargetingKey = "hb_size" HbDealIDConstantKey TargetingKey = "hb_deal" + // HbFormatKey is the format of the bid. For example, "video", "banner" + HbFormatKey TargetingKey = "hb_format" + // HbCacheKey and HbVastCacheKey store UUIDs which can be used to fetch things from prebid cache. // Callers should *never* assume that either of these exist, since the call to the cache may always fail. // diff --git a/openrtb_ext/bid_request_video.go b/openrtb_ext/bid_request_video.go index cbaa47d4f49..13ec8eb4538 100644 --- a/openrtb_ext/bid_request_video.go +++ b/openrtb_ext/bid_request_video.go @@ -144,6 +144,13 @@ type BidRequestVideo struct { // Description: // Indicates that the response should update key to include prefix and tier SupportDeals bool `json:"supportdeals,omitempty"` + + // Attribute: + // appendbiddernames + // Type: + // boolean, optional + // Flag indicating if the bidder name will be added to the hb_pb_cat_dur. Default is false. + AppendBidderNames bool `json:"appendbiddernames,omitempty"` } type PodConfig struct { diff --git a/openrtb_ext/bid_response_video.go b/openrtb_ext/bid_response_video.go index 4c123498ec8..22661547ca7 100644 --- a/openrtb_ext/bid_response_video.go +++ b/openrtb_ext/bid_response_video.go @@ -14,7 +14,7 @@ type AdPod struct { } type VideoTargeting struct { - HbPb string `json:"hb_pb"` - HbPbCatDur string `json:"hb_pb_cat_dur"` - HbCacheID string `json:"hb_cache_id"` + HbPb string `json:"hb_pb,omitempty"` + HbPbCatDur string `json:"hb_pb_cat_dur,omitempty"` + HbCacheID string `json:"hb_cache_id,omitempty"` } diff --git a/openrtb_ext/bidders.go b/openrtb_ext/bidders.go index 424e4c37103..d5c953c57fa 100755 --- a/openrtb_ext/bidders.go +++ b/openrtb_ext/bidders.go @@ -20,30 +20,40 @@ type BidderName string // BidderNameGeneral is reserved for non-bidder specific messages when using a map keyed on the bidder name. const BidderNameGeneral = BidderName("general") +// BidderNameContext is reserved for first party data. +const BidderNameContext = BidderName("context") + // These names _must_ coincide with the bidder code in Prebid.js, if an adapter also exists in that project. // Please keep these (and the BidderMap) alphabetized to minimize merge conflicts among adapter submissions. // The bidder name 'general' is not allowed since it has special meaning in message maps. const ( Bidder33Across BidderName = "33across" + BidderAcuityAds BidderName = "acuityads" BidderAdform BidderName = "adform" BidderAdgeneration BidderName = "adgeneration" BidderAdhese BidderName = "adhese" BidderAdkernel BidderName = "adkernel" BidderAdkernelAdn BidderName = "adkernelAdn" BidderAdpone BidderName = "adpone" + BidderAdman BidderName = "adman" BidderAdmixer BidderName = "admixer" BidderAdOcean BidderName = "adocean" + BidderAdprime BidderName = "adprime" BidderAdtarget BidderName = "adtarget" BidderAdtelligent BidderName = "adtelligent" BidderAdvangelists BidderName = "advangelists" BidderAJA BidderName = "aja" + BidderAMX BidderName = "amx" BidderApplogy BidderName = "applogy" BidderAppnexus BidderName = "appnexus" BidderAdoppler BidderName = "adoppler" BidderAvocet BidderName = "avocet" BidderBeachfront BidderName = "beachfront" BidderBeintoo BidderName = "beintoo" + BidderBetween BidderName = "between" BidderBrightroll BidderName = "brightroll" + BidderColossus BidderName = "colossus" + BidderConnectAd BidderName = "connectad" BidderConsumable BidderName = "consumable" BidderConversant BidderName = "conversant" BidderCpmstar BidderName = "cpmstar" @@ -58,17 +68,22 @@ const ( BidderGrid BidderName = "grid" BidderGumGum BidderName = "gumgum" BidderImprovedigital BidderName = "improvedigital" + BidderInMobi BidderName = "inmobi" + BidderInvibes BidderName = "invibes" BidderIx BidderName = "ix" BidderKidoz BidderName = "kidoz" + BidderKrushmedia BidderName = "krushmedia" BidderKubient BidderName = "kubient" BidderLifestreet BidderName = "lifestreet" BidderLockerDome BidderName = "lockerdome" + BidderLogicad BidderName = "logicad" BidderLunaMedia BidderName = "lunamedia" BidderMarsmedia BidderName = "marsmedia" BidderMgid BidderName = "mgid" BidderMobileFuse BidderName = "mobilefuse" BidderNanoInteractive BidderName = "nanointeractive" BidderNinthDecimal BidderName = "ninthdecimal" + BidderNoBid BidderName = "nobid" BidderOpenx BidderName = "openx" BidderOrbidder BidderName = "orbidder" BidderPubmatic BidderName = "pubmatic" @@ -78,7 +93,11 @@ const ( BidderRTBHouse BidderName = "rtbhouse" BidderRubicon BidderName = "rubicon" BidderSharethrough BidderName = "sharethrough" + BidderSilverMob BidderName = "silvermob" + BidderSmaato BidderName = "smaato" + BidderSmartadserver BidderName = "smartadserver" BidderSmartRTB BidderName = "smartrtb" + BidderSmartyAds BidderName = "smartyads" BidderSomoaudience BidderName = "somoaudience" BidderSonobi BidderName = "sonobi" BidderSovrn BidderName = "sovrn" @@ -105,25 +124,32 @@ const ( // The bidder name 'general' is not allowed since it has special meaning in message maps. var BidderMap = map[string]BidderName{ "33across": Bidder33Across, + "acuityads": BidderAcuityAds, "adform": BidderAdform, "adgeneration": BidderAdgeneration, "adhese": BidderAdhese, "adkernel": BidderAdkernel, "adkernelAdn": BidderAdkernelAdn, + "adman": BidderAdman, "admixer": BidderAdmixer, "adocean": BidderAdOcean, + "adprime": BidderAdprime, "adpone": BidderAdpone, "adtarget": BidderAdtarget, "adtelligent": BidderAdtelligent, "advangelists": BidderAdvangelists, "aja": BidderAJA, + "amx": BidderAMX, "applogy": BidderApplogy, "appnexus": BidderAppnexus, "adoppler": BidderAdoppler, "avocet": BidderAvocet, "beachfront": BidderBeachfront, "beintoo": BidderBeintoo, + "between": BidderBetween, "brightroll": BidderBrightroll, + "colossus": BidderColossus, + "connectad": BidderConnectAd, "consumable": BidderConsumable, "conversant": BidderConversant, "cpmstar": BidderCpmstar, @@ -138,17 +164,22 @@ var BidderMap = map[string]BidderName{ "grid": BidderGrid, "gumgum": BidderGumGum, "improvedigital": BidderImprovedigital, + "inmobi": BidderInMobi, + "invibes": BidderInvibes, "ix": BidderIx, "kidoz": BidderKidoz, + "krushmedia": BidderKrushmedia, "kubient": BidderKubient, "lifestreet": BidderLifestreet, "lockerdome": BidderLockerDome, + "logicad": BidderLogicad, "lunamedia": BidderLunaMedia, "marsmedia": BidderMarsmedia, "mgid": BidderMgid, "mobilefuse": BidderMobileFuse, "nanointeractive": BidderNanoInteractive, "ninthdecimal": BidderNinthDecimal, + "nobid": BidderNoBid, "openx": BidderOpenx, "orbidder": BidderOrbidder, "pubmatic": BidderPubmatic, @@ -158,7 +189,11 @@ var BidderMap = map[string]BidderName{ "rtbhouse": BidderRTBHouse, "rubicon": BidderRubicon, "sharethrough": BidderSharethrough, + "silvermob": BidderSilverMob, + "smaato": BidderSmaato, + "smartadserver": BidderSmartadserver, "smartrtb": BidderSmartRTB, + "smartyads": BidderSmartyAds, "somoaudience": BidderSomoaudience, "sonobi": BidderSonobi, "sovrn": BidderSovrn, diff --git a/openrtb_ext/bidders_test.go b/openrtb_ext/bidders_test.go index d49b23237ed..7b6a03b4de1 100644 --- a/openrtb_ext/bidders_test.go +++ b/openrtb_ext/bidders_test.go @@ -61,3 +61,64 @@ func TestBidderListDoesNotDefineGeneral(t *testing.T) { bidders := BidderList() assert.NotContains(t, bidders, BidderNameGeneral) } + +func TestBidderListDoesNotDefineContext(t *testing.T) { + bidders := BidderList() + assert.NotContains(t, bidders, BidderNameContext) +} + +// TestBidderUniquenessGatekeeping acts as a gatekeeper of bidder name uniqueness. If this test fails +// when you're building a new adapter, please consider choosing a different bidder name to maintain the +// current uniqueness threshold, or else start a discussion in the PR. +func TestBidderUniquenessGatekeeping(t *testing.T) { + // Get List Of Bidders + // - Exclude duplicates of adapters for the same bidder, as it's unlikely a publisher will use both. + var bidders []string + for _, bidder := range BidderMap { + if bidder != BidderTripleliftNative && bidder != BidderAdkernelAdn && bidder != BidderSmartadserver { + bidders = append(bidders, string(bidder)) + } + } + + currentThreshold := 6 + measuredThreshold := minUniquePrefixLength(bidders) + + assert.NotZero(t, measuredThreshold, "BidderMap contains duplicate bidder name values.") + assert.LessOrEqual(t, measuredThreshold, currentThreshold) +} + +// minUniquePrefixLength measures the minimun amount of characters needed to uniquely identify +// one of the strings, or returns 0 if there are duplicates. +func minUniquePrefixLength(b []string) int { + targetingKeyMaxLength := 20 + for prefixLength := 1; prefixLength <= targetingKeyMaxLength; prefixLength++ { + if uniqueForPrefixLength(b, prefixLength) { + return prefixLength + } + } + return 0 +} + +func uniqueForPrefixLength(b []string, prefixLength int) bool { + m := make(map[string]struct{}) + + if prefixLength <= 0 { + return false + } + + for i, n := range b { + ns := string(n) + + if len(ns) > prefixLength { + ns = ns[0:prefixLength] + } + + m[ns] = struct{}{} + + if len(m) != i+1 { + return false + } + } + + return true +} diff --git a/openrtb_ext/deal_tier.go b/openrtb_ext/deal_tier.go new file mode 100644 index 00000000000..e882235d01e --- /dev/null +++ b/openrtb_ext/deal_tier.go @@ -0,0 +1,61 @@ +package openrtb_ext + +import ( + "encoding/json" + + "github.com/PubMatic-OpenWrap/openrtb" +) + +// DealTier defines the configuration of a deal tier. +type DealTier struct { + // Prefix specifies the beginning of the hb_pb_cat_dur targeting key value. Must be non-empty. + Prefix string `json:"prefix"` + + // MinDealTier specifies the minimum deal priority value (inclusive) that must be met for the targeting + // key value to be modified. Must be greater than 0. + MinDealTier int `json:"minDealTier"` +} + +// DealTierBidderMap defines a correlation between bidders and deal tiers. +type DealTierBidderMap map[BidderName]DealTier + +// ReadDealTiersFromImp returns a map of bidder deal tiers read from the impression of an original request (not split / cleaned). +func ReadDealTiersFromImp(imp openrtb.Imp) (DealTierBidderMap, error) { + dealTiers := make(DealTierBidderMap) + + if len(imp.Ext) == 0 { + return dealTiers, nil + } + + // imp.ext.{bidder} + var impExt map[string]struct { + DealTier *DealTier `json:"dealTier"` + } + if err := json.Unmarshal(imp.Ext, &impExt); err != nil { + return nil, err + } + for bidder, param := range impExt { + if param.DealTier != nil { + dealTiers[BidderName(bidder)] = *param.DealTier + } + } + + // imp.ext.prebid.{bidder} + var impPrebidExt struct { + Prebid struct { + Bidders map[string]struct { + DealTier *DealTier `json:"dealTier"` + } `json:"bidder"` + } `json:"prebid"` + } + if err := json.Unmarshal(imp.Ext, &impPrebidExt); err != nil { + return nil, err + } + for bidder, param := range impPrebidExt.Prebid.Bidders { + if param.DealTier != nil { + dealTiers[BidderName(bidder)] = *param.DealTier + } + } + + return dealTiers, nil +} diff --git a/openrtb_ext/deal_tier_test.go b/openrtb_ext/deal_tier_test.go new file mode 100644 index 00000000000..717e0703466 --- /dev/null +++ b/openrtb_ext/deal_tier_test.go @@ -0,0 +1,98 @@ +package openrtb_ext + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/stretchr/testify/assert" +) + +func TestReadDealTiersFromImp(t *testing.T) { + testCases := []struct { + description string + impExt json.RawMessage + expectedResult DealTierBidderMap + expectedError string + }{ + { + description: "Nil", + impExt: nil, + expectedResult: DealTierBidderMap{}, + }, + { + description: "None", + impExt: json.RawMessage(``), + expectedResult: DealTierBidderMap{}, + }, + { + description: "Empty Object", + impExt: json.RawMessage(`{}`), + expectedResult: DealTierBidderMap{}, + }, + { + description: "imp.ext - with other params", + impExt: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "anyPrefix"}, "placementId": 12345}}`), + expectedResult: DealTierBidderMap{BidderAppnexus: {Prefix: "anyPrefix", MinDealTier: 5}}, + }, + { + description: "imp.ext - multiple", + impExt: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "appnexusPrefix"}, "placementId": 12345}, "rubicon": {"dealTier": {"minDealTier": 8, "prefix": "rubiconPrefix"}, "placementId": 12345}}`), + expectedResult: DealTierBidderMap{BidderAppnexus: {Prefix: "appnexusPrefix", MinDealTier: 5}, BidderRubicon: {Prefix: "rubiconPrefix", MinDealTier: 8}}, + }, + { + description: "imp.ext - no deal tier", + impExt: json.RawMessage(`{"appnexus": {"placementId": 12345}}`), + expectedResult: DealTierBidderMap{}, + }, + { + description: "imp.ext - error", + impExt: json.RawMessage(`{"appnexus": {"dealTier": "wrong type", "placementId": 12345}}`), + expectedError: "json: cannot unmarshal string into Go struct field .dealTier of type openrtb_ext.DealTier", + }, + { + description: "imp.ext.prebid", + impExt: json.RawMessage(`{"prebid": {"bidder": {"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "anyPrefix"}, "placementId": 12345}}}}`), + expectedResult: DealTierBidderMap{BidderAppnexus: {Prefix: "anyPrefix", MinDealTier: 5}}, + }, + { + description: "imp.ext.prebid- multiple", + impExt: json.RawMessage(`{"prebid": {"bidder": {"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "appnexusPrefix"}, "placementId": 12345}, "rubicon": {"dealTier": {"minDealTier": 8, "prefix": "rubiconPrefix"}, "placementId": 12345}}}}`), + expectedResult: DealTierBidderMap{BidderAppnexus: {Prefix: "appnexusPrefix", MinDealTier: 5}, BidderRubicon: {Prefix: "rubiconPrefix", MinDealTier: 8}}, + }, + { + description: "imp.ext.prebid - no deal tier", + impExt: json.RawMessage(`{"prebid": {"bidder": {"appnexus": {"placementId": 12345}}}}`), + expectedResult: DealTierBidderMap{}, + }, + { + description: "imp.ext.prebid - error", + impExt: json.RawMessage(`{"prebid": {"bidder": {"appnexus": {"dealTier": "wrong type", "placementId": 12345}}}}`), + expectedError: "json: cannot unmarshal string into Go struct field .prebid.bidder.dealTier of type openrtb_ext.DealTier", + }, + { + description: "imp.ext.prebid wins over imp.ext", + impExt: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "impExt"}, "placementId": 12345}, "prebid": {"bidder": {"appnexus": {"dealTier": {"minDealTier": 8, "prefix": "impExtPrebid"}, "placementId": 12345}}}}`), + expectedResult: DealTierBidderMap{BidderAppnexus: {Prefix: "impExtPrebid", MinDealTier: 8}}, + }, + { + description: "imp.ext.prebid coexists with imp.ext", + impExt: json.RawMessage(`{"appnexus": {"dealTier": {"minDealTier": 5, "prefix": "impExt"}, "placementId": 12345}, "prebid": {"bidder": {"rubicon": {"dealTier": {"minDealTier": 8, "prefix": "impExtPrebid"}, "placementId": 12345}}}}`), + expectedResult: DealTierBidderMap{BidderAppnexus: {Prefix: "impExt", MinDealTier: 5}, BidderRubicon: {Prefix: "impExtPrebid", MinDealTier: 8}}, + }, + } + + for _, test := range testCases { + imp := openrtb.Imp{Ext: test.impExt} + + result, err := ReadDealTiersFromImp(imp) + + assert.Equal(t, test.expectedResult, result, test.description+":result") + + if len(test.expectedError) == 0 { + assert.NoError(t, err, test.description+":error") + } else { + assert.EqualError(t, err, test.expectedError, test.description+":error") + } + } +} diff --git a/openrtb_ext/imp.go b/openrtb_ext/imp.go index 0d5e1f655cb..f83fa63df84 100644 --- a/openrtb_ext/imp.go +++ b/openrtb_ext/imp.go @@ -4,30 +4,16 @@ import ( "encoding/json" ) -// ExtImp defines the contract for bidrequest.imp[i].ext -type ExtImp struct { - Prebid *ExtImpPrebid `json:"prebid,omitempty"` - Appnexus *ExtImpAppnexus `json:"appnexus"` - Consumable *ExtImpConsumable `json:"consumable"` - Rubicon *ExtImpRubicon `json:"rubicon"` - Adform *ExtImpAdform `json:"adform"` - Rhythmone *ExtImpRhythmone `json:"rhythmone"` - Unruly *ExtImpUnruly `json:"unruly"` - EmxDigital *ExtImpEmxDigital `json:"emx_digital"` -} - // ExtImpPrebid defines the contract for bidrequest.imp[i].ext.prebid type ExtImpPrebid struct { - StoredRequest *ExtStoredRequest `json:"storedrequest,omitempty"` + // StoredRequest specifies which stored impression to use, if any. + StoredRequest *ExtStoredRequest `json:"storedrequest"` - // Rewarded inventory signal, can be 0 or 1 - IsRewardedInventory int8 `json:"is_rewarded_inventory,omitempty"` + // IsRewardedInventory is a signal intended for video impressions. Must be 0 or 1. + IsRewardedInventory int8 `json:"is_rewarded_inventory"` - // NOTE: This is not part of the official API, we are not expecting clients - // migrate from imp[...].ext.${BIDDER} to imp[...].ext.prebid.bidder.${BIDDER} - // at this time - // https://github.com/PubMatic-OpenWrap/prebid-server/pull/846#issuecomment-476352224 - Bidder map[string]json.RawMessage `json:"bidder,omitempty"` + // Bidder is the preferred approach for providing paramters to be interepreted by the bidder's adapter. + Bidder map[string]json.RawMessage `json:"bidder"` SKAdnetwork json.RawMessage `json:"skadn,omitempty"` } diff --git a/openrtb_ext/imp_acuityads.go b/openrtb_ext/imp_acuityads.go new file mode 100644 index 00000000000..f0275e39f89 --- /dev/null +++ b/openrtb_ext/imp_acuityads.go @@ -0,0 +1,6 @@ +package openrtb_ext + +type ExtAcuityAds struct { + Host string `json:"host"` + AccountID string `json:"accountid"` +} diff --git a/openrtb_ext/imp_adform.go b/openrtb_ext/imp_adform.go index 3e7c1a7261e..3206ece7c9b 100644 --- a/openrtb_ext/imp_adform.go +++ b/openrtb_ext/imp_adform.go @@ -1,8 +1,11 @@ package openrtb_ext type ExtImpAdform struct { - MasterTagId string `json:"mid"` - PriceType string `json:"priceType,omitempty"` - KeyValues string `json:"mkv,omitempty"` - KeyWords string `json:"mkw,omitempty"` + MasterTagId string `json:"mid"` + PriceType string `json:"priceType,omitempty"` + KeyValues string `json:"mkv,omitempty"` + KeyWords string `json:"mkw,omitempty"` + CDims string `json:"cdims,omitempty"` + MinPrice float64 `json:"minp,omitempty"` + Url string `json:"url,omitempty"` } diff --git a/openrtb_ext/imp_adman.go b/openrtb_ext/imp_adman.go new file mode 100644 index 00000000000..bc79415452c --- /dev/null +++ b/openrtb_ext/imp_adman.go @@ -0,0 +1,6 @@ +package openrtb_ext + +// ExtImpAdman defines adman specifiec param +type ExtImpAdman struct { + TagID string `json:"TagID"` +} diff --git a/openrtb_ext/imp_adoppler.go b/openrtb_ext/imp_adoppler.go index 4b3ba97ce05..9d4d5e5ca01 100644 --- a/openrtb_ext/imp_adoppler.go +++ b/openrtb_ext/imp_adoppler.go @@ -1,5 +1,6 @@ package openrtb_ext type ExtImpAdoppler struct { + Client string `json:"client"` AdUnit string `json:"adunit"` } diff --git a/openrtb_ext/imp_adprime.go b/openrtb_ext/imp_adprime.go new file mode 100644 index 00000000000..a089b818b56 --- /dev/null +++ b/openrtb_ext/imp_adprime.go @@ -0,0 +1,6 @@ +package openrtb_ext + +// ExtImpAdprime defines adprime specifiec param +type ExtImpAdprime struct { + TagID string `json:"TagID"` +} diff --git a/openrtb_ext/imp_amx.go b/openrtb_ext/imp_amx.go new file mode 100644 index 00000000000..d4439d05f60 --- /dev/null +++ b/openrtb_ext/imp_amx.go @@ -0,0 +1,7 @@ +package openrtb_ext + +// ExtImpAMX is the imp.ext format for the AMX bidder +type ExtImpAMX struct { + TagID string `json:"tagId,omitempty"` + AdUnitID string `json:"adUnitId,omitempty"` +} diff --git a/openrtb_ext/imp_between.go b/openrtb_ext/imp_between.go new file mode 100644 index 00000000000..788ce215b9a --- /dev/null +++ b/openrtb_ext/imp_between.go @@ -0,0 +1,5 @@ +package openrtb_ext + +type ExtImpBetween struct { + Host string `json:"host"` +} diff --git a/openrtb_ext/imp_colossus.go b/openrtb_ext/imp_colossus.go new file mode 100644 index 00000000000..8969000558d --- /dev/null +++ b/openrtb_ext/imp_colossus.go @@ -0,0 +1,6 @@ +package openrtb_ext + +// ExtImpColossus defines colossus specifiec param +type ExtImpColossus struct { + TagID string `json:"TagID"` +} diff --git a/openrtb_ext/imp_connectad.go b/openrtb_ext/imp_connectad.go new file mode 100644 index 00000000000..c4c7ab696f2 --- /dev/null +++ b/openrtb_ext/imp_connectad.go @@ -0,0 +1,7 @@ +package openrtb_ext + +type ExtImpConnectAd struct { + NetworkID int `json:"networkId"` + SiteID int `json:"siteId"` + Bidfloor float64 `json:"bidfloor,omitempty"` +} diff --git a/openrtb_ext/imp_conversant.go b/openrtb_ext/imp_conversant.go new file mode 100644 index 00000000000..8587e111153 --- /dev/null +++ b/openrtb_ext/imp_conversant.go @@ -0,0 +1,13 @@ +package openrtb_ext + +type ExtImpConversant struct { + SiteID string `json:"site_id"` + Secure *int8 `json:"secure"` + TagID string `json:"tag_id"` + Position *int8 `json:"position"` + BidFloor float64 `json:"bidfloor"` + MIMEs []string `json:"mimes"` + API []int8 `json:"api"` + Protocols []int8 `json:"protocols"` + MaxDuration *int64 `json:"maxduration"` +} diff --git a/openrtb_ext/imp_inmobi.go b/openrtb_ext/imp_inmobi.go new file mode 100644 index 00000000000..d74e3cac8b0 --- /dev/null +++ b/openrtb_ext/imp_inmobi.go @@ -0,0 +1,5 @@ +package openrtb_ext + +type ExtImpInMobi struct { + Plc string `json:"plc"` +} diff --git a/openrtb_ext/imp_invibes.go b/openrtb_ext/imp_invibes.go new file mode 100644 index 00000000000..37ed31ced63 --- /dev/null +++ b/openrtb_ext/imp_invibes.go @@ -0,0 +1,12 @@ +package openrtb_ext + +type ExtImpInvibes struct { + PlacementID string `json:"placementId,omitempty"` + DomainID int `json:"domainId"` + Debug ExtImpInvibesDebug `json:"debug,omitempty"` +} + +type ExtImpInvibesDebug struct { + TestBvid string `json:"testBvid,omitempty"` + TestLog bool `json:"testLog,omitempty"` +} diff --git a/openrtb_ext/imp_krushmedia.go b/openrtb_ext/imp_krushmedia.go new file mode 100644 index 00000000000..a175c227fda --- /dev/null +++ b/openrtb_ext/imp_krushmedia.go @@ -0,0 +1,6 @@ +package openrtb_ext + +// ExtKrushmedia defines imp[0].ext object structure +type ExtKrushmedia struct { + AccountID string `json:"key"` +} diff --git a/openrtb_ext/imp_kubient.go b/openrtb_ext/imp_kubient.go new file mode 100644 index 00000000000..fafd2a0eb8f --- /dev/null +++ b/openrtb_ext/imp_kubient.go @@ -0,0 +1,6 @@ +package openrtb_ext + +// ExtImpKubient defines the contract for bidrequest.imp[i].ext.kubient +type ExtImpKubient struct { + ZoneID string `json:"zoneid"` +} diff --git a/openrtb_ext/imp_logicad.go b/openrtb_ext/imp_logicad.go new file mode 100644 index 00000000000..e4e3c3b091c --- /dev/null +++ b/openrtb_ext/imp_logicad.go @@ -0,0 +1,5 @@ +package openrtb_ext + +type ExtImpLogicad struct { + Tid string `json:"tid"` +} diff --git a/openrtb_ext/imp_nobid.go b/openrtb_ext/imp_nobid.go new file mode 100644 index 00000000000..8af16952c39 --- /dev/null +++ b/openrtb_ext/imp_nobid.go @@ -0,0 +1,6 @@ +package openrtb_ext + +type ExtImpNoBid struct { + SiteID string `json:"siteId"` + PlacementID string `json:"placementId"` +} diff --git a/openrtb_ext/imp_openx.go b/openrtb_ext/imp_openx.go index e63595b0912..2625cb3802d 100644 --- a/openrtb_ext/imp_openx.go +++ b/openrtb_ext/imp_openx.go @@ -3,6 +3,7 @@ package openrtb_ext // ExtImpOpenx defines the contract for bidrequest.imp[i].ext.openx type ExtImpOpenx struct { Unit string `json:"unit"` + Platform string `json:"platform"` DelDomain string `json:"delDomain"` CustomFloor float64 `json:"customFloor"` CustomParams map[string]interface{} `json:"customParams"` diff --git a/openrtb_ext/imp_silvermob.go b/openrtb_ext/imp_silvermob.go new file mode 100644 index 00000000000..9b2465534ca --- /dev/null +++ b/openrtb_ext/imp_silvermob.go @@ -0,0 +1,7 @@ +package openrtb_ext + +// ExtSilverMob defines the contract for bidrequest.imp[i].ext.silvermob +type ExtSilverMob struct { + ZoneID string `json:"zoneid"` + Host string `json:"host"` +} diff --git a/openrtb_ext/imp_smaato.go b/openrtb_ext/imp_smaato.go new file mode 100644 index 00000000000..10de97fb017 --- /dev/null +++ b/openrtb_ext/imp_smaato.go @@ -0,0 +1,9 @@ +package openrtb_ext + +// ExtImpSmaato defines the contract for bidrequest.imp[i].ext.smaato +// PublisherId and AdSpaceId are mandatory parameters, others are optional parameters +// AdSpaceId is identifier for specific ad placement or ad tag +type ExtImpSmaato struct { + PublisherID string `json:"publisherId"` + AdSpaceID string `json:"adspaceId"` +} diff --git a/openrtb_ext/imp_smartadserver.go b/openrtb_ext/imp_smartadserver.go new file mode 100644 index 00000000000..d542e0ffd27 --- /dev/null +++ b/openrtb_ext/imp_smartadserver.go @@ -0,0 +1,9 @@ +package openrtb_ext + +// ExtImpSmartadserver defines the contract for bidrequest.imp[i].ext.smartadserver +type ExtImpSmartadserver struct { + SiteID int `json:"siteId"` + PageID int `json:"pageId"` + FormatID int `json:"formatId"` + NetworkID int `json:"networkId"` +} diff --git a/openrtb_ext/imp_smartyads.go b/openrtb_ext/imp_smartyads.go new file mode 100644 index 00000000000..54911373e61 --- /dev/null +++ b/openrtb_ext/imp_smartyads.go @@ -0,0 +1,8 @@ +package openrtb_ext + +// ExtSmartyAds defines the contract for bidrequest.imp[i].ext.smartyads +type ExtSmartyAds struct { + AccountID string `json:"accountid"` + SourceID string `json:"sourceid"` + Host string `json:"host"` +} diff --git a/openrtb_ext/imp_telaria.go b/openrtb_ext/imp_telaria.go index 8ea371a8ad0..19a025c0b15 100644 --- a/openrtb_ext/imp_telaria.go +++ b/openrtb_ext/imp_telaria.go @@ -1,6 +1,9 @@ package openrtb_ext +import "encoding/json" + type ExtImpTelaria struct { - AdCode string `json:"adCode,omitempty"` - SeatCode string `json:"seatCode"` + AdCode string `json:"adCode,omitempty"` + SeatCode string `json:"seatCode"` + Extra json.RawMessage `json:"extra,omitempty"` } diff --git a/openrtb_ext/request.go b/openrtb_ext/request.go index ca7c0b40c17..e3684dffa5f 100644 --- a/openrtb_ext/request.go +++ b/openrtb_ext/request.go @@ -5,6 +5,11 @@ import ( "errors" ) +// FirstPartyDataContextExtKey defines the field name within bidrequest.ext reserved +// for first party data support. +const FirstPartyDataContextExtKey string = "context" +const MaxDecimalFigures int = 15 + // ExtRequest defines the contract for bidrequest.ext type ExtRequest struct { Prebid ExtRequestPrebid `json:"prebid"` @@ -12,14 +17,50 @@ type ExtRequest struct { // ExtRequestPrebid defines the contract for bidrequest.ext.prebid type ExtRequestPrebid struct { - Aliases map[string]string `json:"aliases,omitempty"` - BidAdjustmentFactors map[string]float64 `json:"bidadjustmentfactors,omitempty"` - Cache *ExtRequestPrebidCache `json:"cache,omitempty"` - StoredRequest *ExtStoredRequest `json:"storedrequest,omitempty"` - Targeting *ExtRequestTargeting `json:"targeting,omitempty"` - SupportDeals bool `json:"supportdeals,omitempty"` - Debug int `json:"debug,omitempty"` - BidderParams interface{} `json:"bidderparams,omitempty"` + Aliases map[string]string `json:"aliases,omitempty"` + BidAdjustmentFactors map[string]float64 `json:"bidadjustmentfactors,omitempty"` + Cache *ExtRequestPrebidCache `json:"cache,omitempty"` + SChains []*ExtRequestPrebidSChain `json:"schains,omitempty"` + StoredRequest *ExtStoredRequest `json:"storedrequest,omitempty"` + Targeting *ExtRequestTargeting `json:"targeting,omitempty"` + SupportDeals bool `json:"supportdeals,omitempty"` + Debug bool `json:"debug,omitempty"` + BidderParams interface{} `json:"bidderparams,omitempty"` + + // NoSale specifies bidders with whom the publisher has a legal relationship where the + // passing of personally identifiable information doesn't constitute a sale per CCPA law. + // The array may contain a single sstar ('*') entry to represent all bidders. + NoSale []string `json:"nosale,omitempty"` +} + +// ExtRequestPrebid defines the contract for bidrequest.ext.prebid.schains +type ExtRequestPrebidSChain struct { + Bidders []string `json:"bidders,omitempty"` + SChain ExtRequestPrebidSChainSChain `json:"schain"` +} + +// ExtRequestPrebidSChainSChain defines the contract for bidrequest.ext.prebid.schains[i].schain +type ExtRequestPrebidSChainSChain struct { + Complete int `json:"complete"` + Nodes []*ExtRequestPrebidSChainSChainNode `json:"nodes"` + Ver string `json:"ver"` + Ext json.RawMessage `json:"ext,omitempty"` +} + +// ExtRequestPrebidSChainSChainNode defines the contract for bidrequest.ext.prebid.schains[i].schain[i].nodes +type ExtRequestPrebidSChainSChainNode struct { + ASI string `json:"asi"` + SID string `json:"sid"` + RID string `json:"rid,omitempty"` + Name string `json:"name,omitempty"` + Domain string `json:"domain,omitempty"` + HP int `json:"hp"` + Ext json.RawMessage `json:"ext,omitempty"` +} + +// SourceExt defines the contract for bidrequest.source.ext +type SourceExt struct { + SChain ExtRequestPrebidSChainSChain `json:"schain"` } // ExtRequestPrebidCache defines the contract for bidrequest.ext.prebid.cache @@ -37,7 +78,7 @@ func (ert *ExtRequestPrebidCache) UnmarshalJSON(b []byte) error { } if proxy.Bids == nil && proxy.VastXML == nil { - return errors.New(`request.ext.prebid.cache requires one of the "bids" or "vastml" properties`) + return errors.New(`request.ext.prebid.cache requires one of the "bids" or "vastxml" properties`) } *ert = ExtRequestPrebidCache(proxy) @@ -45,10 +86,14 @@ func (ert *ExtRequestPrebidCache) UnmarshalJSON(b []byte) error { } // ExtRequestPrebidCacheBids defines the contract for bidrequest.ext.prebid.cache.bids -type ExtRequestPrebidCacheBids struct{} +type ExtRequestPrebidCacheBids struct { + ReturnCreative *bool `json:"returnCreative"` +} // ExtRequestPrebidCacheVAST defines the contract for bidrequest.ext.prebid.cache.vastxml -type ExtRequestPrebidCacheVAST struct{} +type ExtRequestPrebidCacheVAST struct { + ReturnCreative *bool `json:"returnCreative"` +} // ExtRequestTargeting defines the contract for bidrequest.ext.prebid.targeting type ExtRequestTargeting struct { @@ -56,7 +101,10 @@ type ExtRequestTargeting struct { IncludeWinners bool `json:"includewinners"` IncludeBidderKeys bool `json:"includebidderkeys"` IncludeBrandCategory *ExtIncludeBrandCategory `json:"includebrandcategory"` + IncludeFormat bool `json:"includeformat"` DurationRangeSec []int `json:"durationrangesec"` + PreferDeals bool `json:"preferdeals"` + AppendBidderNames bool `json:"appendbiddernames,omitempty"` } type ExtIncludeBrandCategory struct { @@ -135,6 +183,9 @@ func (pg *PriceGranularity) UnmarshalJSON(b []byte) error { if pgraw.Precision < 0 { return errors.New("Price granularity error: precision must be non-negative") } + if pgraw.Precision > MaxDecimalFigures { + return errors.New("Price granularity error: precision of more than 15 significant figures is not supported") + } if len(pgraw.Ranges) > 0 { var prevMax float64 = 0 for i, gr := range pgraw.Ranges { @@ -146,9 +197,6 @@ func (pg *PriceGranularity) UnmarshalJSON(b []byte) error { } // Enforce that we don't read "min" from the request pgraw.Ranges[i].Min = prevMax - if pgraw.Ranges[i].Min < prevMax { - return errors.New("Price granularity error: overlapping granularity ranges") - } prevMax = gr.Max } *pg = PriceGranularity(pgraw) diff --git a/openrtb_ext/request_test.go b/openrtb_ext/request_test.go index e4046a622db..98a2e1645a0 100644 --- a/openrtb_ext/request_test.go +++ b/openrtb_ext/request_test.go @@ -190,21 +190,43 @@ var validGranularityTests []granularityTestData = []granularityTestData{ } func TestGranularityUnmarshalBad(t *testing.T) { - tests := [][]byte{ - []byte(`[]`), - []byte(`{"precision": -1, "ranges": [{"max":20, "increment":0.5}]}`), - []byte(`{"ranges":[{"max":20, "increment": -1}]}`), - []byte(`{"ranges":[{"max":"20", "increment": "0.1"}]}`), - []byte(`{"ranges":[{"max":20, "increment":0.1}. {"max":10, "increment":0.02}]}`), - []byte(`{"ranges":[{"max":20, "min":10, "increment": 0.1}, {"max":10, "min":0, "increment":0.05}]}`), - []byte(`{"ranges":[{"max":1.0, "increment": 0.07}, {"max" 1.0, "increment": 0.03}]}`), + testCases := []struct { + description string + jsonPriceGranularity []byte + }{ + { + "Malformed", + []byte(`[]`), + }, + { + "Negative precision", + []byte(`{"precision": -1, "ranges": [{"max":20, "increment":0.5}]}`), + }, + { + "Precision greater than MaxDecimalFigures supported", + []byte(`{"precision": 16, "ranges": [{"max":20, "increment":0.5}]}`), + }, + { + "Negative increment", + []byte(`{"ranges":[{"max":20, "increment": -1}]}`), + }, + { + "Range with non float64 max value", + []byte(`{"ranges":[{"max":"20", "increment": "0.1"}]}`), + }, + { + "Ranges in decreasing order", + []byte(`{"ranges":[{"max":20, "increment":0.1}. {"max":10, "increment":0.02}]}`), + }, + { + "Max equal to previous max", + []byte(`{"ranges":[{"max":1.0, "increment": 0.07}, {"max" 1.0, "increment": 0.03}]}`), + }, } - var resolved PriceGranularity - for _, b := range tests { - resolved = PriceGranularity{} - err := json.Unmarshal(b, &resolved) - if err == nil { - t.Errorf("Invalid granularity unmarshalled without error.\nJSON was: %s\n Resolved to: %v", string(b), resolved) - } + + for _, test := range testCases { + resolved := PriceGranularity{} + err := json.Unmarshal(test.jsonPriceGranularity, &resolved) + assert.Errorf(t, err, "Invalid granularity unmarshalled without error.\nJSON was: %s\n Resolved to: %v. Test: %s", string(test.jsonPriceGranularity), resolved, test.description) } } diff --git a/openrtb_ext/user.go b/openrtb_ext/user.go index a7c8505d226..b83f82330db 100644 --- a/openrtb_ext/user.go +++ b/openrtb_ext/user.go @@ -33,7 +33,7 @@ type ExtUserDigiTrust struct { // ExtUserEid defines the contract for bidrequest.user.ext.eids // Responsible for the Universal User ID support: establishing pseudonymous IDs for users. -// See https://github.com/PubMatic-OpenWrap/Prebid.js/issues/3900 for details. +// See https://github.com/prebid/Prebid.js/issues/3900 for details. type ExtUserEid struct { Source string `json:"source"` ID string `json:"id,omitempty"` @@ -44,6 +44,6 @@ type ExtUserEid struct { // ExtUserEidUid defines the contract for bidrequest.user.ext.eids[i].uids[j] type ExtUserEidUid struct { ID string `json:"id"` - AType int `json:"atype,omitempty"` + Atype int `json:"atype,omitempty"` Ext json.RawMessage `json:"ext,omitempty"` } diff --git a/pbs/pbsrequest.go b/pbs/pbsrequest.go index bb8db11ba90..bd07a6c558b 100644 --- a/pbs/pbsrequest.go +++ b/pbs/pbsrequest.go @@ -12,9 +12,10 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/cache" "github.com/PubMatic-OpenWrap/prebid-server/config" - "github.com/PubMatic-OpenWrap/prebid-server/prebid" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/usersync" + "github.com/PubMatic-OpenWrap/prebid-server/util/httputil" + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" "github.com/PubMatic-OpenWrap/openrtb" "github.com/blang/semver" @@ -216,6 +217,8 @@ func ParseMediaTypes(types []string) []MediaType { return mtypes } +var ipv4Validator iputil.IPValidator = iputil.VersionIPValidator{iputil.IPv4} + func ParsePBSRequest(r *http.Request, cfg *config.AuctionTimeouts, cache cache.Cache, hostCookieConfig *config.HostCookie) (*PBSRequest, error) { defer r.Body.Close() @@ -235,7 +238,9 @@ func ParsePBSRequest(r *http.Request, cfg *config.AuctionTimeouts, cache cache.C if pbsReq.Device == nil { pbsReq.Device = &openrtb.Device{} } - pbsReq.Device.IP = prebid.GetIP(r) + if ip, _ := httputil.FindIP(r, ipv4Validator); ip != nil { + pbsReq.Device.IP = ip.String() + } if pbsReq.SDK == nil { pbsReq.SDK = &SDK{} @@ -291,7 +296,7 @@ func ParsePBSRequest(r *http.Request, cfg *config.AuctionTimeouts, cache cache.C pbsReq.IsDebug = true } - if prebid.IsSecure(r) { + if httputil.IsSecure(r) { pbsReq.Secure = 1 } diff --git a/pbsmetrics/config/metrics.go b/pbsmetrics/config/metrics.go index e275910fa6e..10eda83a856 100644 --- a/pbsmetrics/config/metrics.go +++ b/pbsmetrics/config/metrics.go @@ -37,7 +37,7 @@ func NewMetricsEngine(cfg *config.Configuration, adapterList []openrtb_ext.Bidde } if cfg.Metrics.Prometheus.Port != 0 { // Set up the Prometheus metrics. - returnEngine.PrometheusMetrics = prometheusmetrics.NewMetrics(cfg.Metrics.Prometheus) + returnEngine.PrometheusMetrics = prometheusmetrics.NewMetrics(cfg.Metrics.Prometheus, cfg.Metrics.Disabled) engineList = append(engineList, returnEngine.PrometheusMetrics) } @@ -104,6 +104,20 @@ func (me *MultiMetricsEngine) RecordRequestTime(labels pbsmetrics.Labels, length } } +// RecordStoredDataFetchTime across all engines +func (me *MultiMetricsEngine) RecordStoredDataFetchTime(labels pbsmetrics.StoredDataLabels, length time.Duration) { + for _, thisME := range *me { + thisME.RecordStoredDataFetchTime(labels, length) + } +} + +// RecordStoredDataError across all engines +func (me *MultiMetricsEngine) RecordStoredDataError(labels pbsmetrics.StoredDataLabels) { + for _, thisME := range *me { + thisME.RecordStoredDataError(labels) + } +} + // RecordAdapterPanic across all engines func (me *MultiMetricsEngine) RecordAdapterPanic(labels pbsmetrics.AdapterLabels) { for _, thisME := range *me { @@ -118,6 +132,21 @@ func (me *MultiMetricsEngine) RecordAdapterRequest(labels pbsmetrics.AdapterLabe } } +// Keeps track of created and reused connections to adapter bidders and the time from the +// connection request, to the connection creation, or reuse from the pool across all engines +func (me *MultiMetricsEngine) RecordAdapterConnections(bidderName openrtb_ext.BidderName, connWasReused bool, connWaitTime time.Duration) { + for _, thisME := range *me { + thisME.RecordAdapterConnections(bidderName, connWasReused, connWaitTime) + } +} + +// Times the DNS resolution process +func (me *MultiMetricsEngine) RecordDNSTime(dnsLookupTime time.Duration) { + for _, thisME := range *me { + thisME.RecordDNSTime(dnsLookupTime) + } +} + // RecordAdapterBidReceived across all engines func (me *MultiMetricsEngine) RecordAdapterBidReceived(labels pbsmetrics.AdapterLabels, bidType openrtb_ext.BidType, hasAdm bool) { for _, thisME := range *me { @@ -160,6 +189,13 @@ func (me *MultiMetricsEngine) RecordStoredImpCacheResult(cacheResult pbsmetrics. } } +// RecordAccountCacheResult across all engines +func (me *MultiMetricsEngine) RecordAccountCacheResult(cacheResult pbsmetrics.CacheResult, inc int) { + for _, thisME := range *me { + thisME.RecordAccountCacheResult(cacheResult, inc) + } +} + // RecordAdapterCookieSync across all engines func (me *MultiMetricsEngine) RecordAdapterCookieSync(adapter openrtb_ext.BidderName, gdprBlocked bool) { for _, thisME := range *me { @@ -195,6 +231,13 @@ func (me *MultiMetricsEngine) RecordTimeoutNotice(success bool) { } } +// RecordRequestPrivacy across all engines +func (me *MultiMetricsEngine) RecordRequestPrivacy(privacy pbsmetrics.PrivacyLabels) { + for _, thisME := range *me { + thisME.RecordRequestPrivacy(privacy) + } +} + // RecordAdapterDuplicateBidID across all engines func (me *MultiMetricsEngine) RecordAdapterDuplicateBidID(adaptor string, collisions int) { for _, thisME := range *me { @@ -264,6 +307,14 @@ func (me *DummyMetricsEngine) RecordLegacyImps(labels pbsmetrics.Labels, numImps func (me *DummyMetricsEngine) RecordRequestTime(labels pbsmetrics.Labels, length time.Duration) { } +// RecordStoredDataFetchTime as a noop +func (me *DummyMetricsEngine) RecordStoredDataFetchTime(labels pbsmetrics.StoredDataLabels, length time.Duration) { +} + +// RecordStoredDataError as a noop +func (me *DummyMetricsEngine) RecordStoredDataError(labels pbsmetrics.StoredDataLabels) { +} + // RecordAdapterPanic as a noop func (me *DummyMetricsEngine) RecordAdapterPanic(labels pbsmetrics.AdapterLabels) { } @@ -272,6 +323,14 @@ func (me *DummyMetricsEngine) RecordAdapterPanic(labels pbsmetrics.AdapterLabels func (me *DummyMetricsEngine) RecordAdapterRequest(labels pbsmetrics.AdapterLabels) { } +// RecordAdapterConnections as a noop +func (me *DummyMetricsEngine) RecordAdapterConnections(bidderName openrtb_ext.BidderName, connWasReused bool, connWaitTime time.Duration) { +} + +// RecordDNSTime as a noop +func (me *DummyMetricsEngine) RecordDNSTime(dnsLookupTime time.Duration) { +} + // RecordAdapterBidReceived as a noop func (me *DummyMetricsEngine) RecordAdapterBidReceived(labels pbsmetrics.AdapterLabels, bidType openrtb_ext.BidType, hasAdm bool) { } @@ -304,6 +363,10 @@ func (me *DummyMetricsEngine) RecordStoredReqCacheResult(cacheResult pbsmetrics. func (me *DummyMetricsEngine) RecordStoredImpCacheResult(cacheResult pbsmetrics.CacheResult, inc int) { } +// RecordAccountCacheResult as a noop +func (me *DummyMetricsEngine) RecordAccountCacheResult(cacheResult pbsmetrics.CacheResult, inc int) { +} + // RecordPrebidCacheRequestTime as a noop func (me *DummyMetricsEngine) RecordPrebidCacheRequestTime(success bool, length time.Duration) { } @@ -316,6 +379,10 @@ func (me *DummyMetricsEngine) RecordRequestQueueTime(success bool, requestType p func (me *DummyMetricsEngine) RecordTimeoutNotice(success bool) { } +// RecordRequestPrivacy as a noop +func (me *DummyMetricsEngine) RecordRequestPrivacy(privacy pbsmetrics.PrivacyLabels) { +} + // RecordAdapterDuplicateBidID as a noop func (me *DummyMetricsEngine) RecordAdapterDuplicateBidID(adaptor string, collisions int) { } diff --git a/pbsmetrics/config/metrics_test.go b/pbsmetrics/config/metrics_test.go index 26635569969..288a9e6ff11 100644 --- a/pbsmetrics/config/metrics_test.go +++ b/pbsmetrics/config/metrics_test.go @@ -116,6 +116,13 @@ func TestMultiMetricsEngine(t *testing.T) { metricsEngine.RecordImps(impTypeLabels) } + metricsEngine.RecordStoredReqCacheResult(pbsmetrics.CacheMiss, 1) + metricsEngine.RecordStoredImpCacheResult(pbsmetrics.CacheMiss, 2) + metricsEngine.RecordAccountCacheResult(pbsmetrics.CacheMiss, 3) + metricsEngine.RecordStoredReqCacheResult(pbsmetrics.CacheHit, 4) + metricsEngine.RecordStoredImpCacheResult(pbsmetrics.CacheHit, 5) + metricsEngine.RecordAccountCacheResult(pbsmetrics.CacheHit, 6) + metricsEngine.RecordRequestQueueTime(false, pbsmetrics.ReqTypeVideo, time.Duration(1)) //Make the metrics engine, instantiated here with goEngine, fill its RequestStatuses[RequestType][pbsmetrics.RequestStatusXX] with the new boolean values added to pbsmetrics.Labels @@ -154,6 +161,13 @@ func TestMultiMetricsEngine(t *testing.T) { VerifyMetrics(t, "RecordRequestQueueTime.Video.Rejected", goEngine.RequestsQueueTimer[pbsmetrics.ReqTypeVideo][false].Count(), 1) VerifyMetrics(t, "RecordRequestQueueTime.Video.Accepted", goEngine.RequestsQueueTimer[pbsmetrics.ReqTypeVideo][true].Count(), 0) + + VerifyMetrics(t, "StoredReqCache.Miss", goEngine.StoredReqCacheMeter[pbsmetrics.CacheMiss].Count(), 1) + VerifyMetrics(t, "StoredImpCache.Miss", goEngine.StoredImpCacheMeter[pbsmetrics.CacheMiss].Count(), 2) + VerifyMetrics(t, "AccountCache.Miss", goEngine.AccountCacheMeter[pbsmetrics.CacheMiss].Count(), 3) + VerifyMetrics(t, "StoredReqCache.Hit", goEngine.StoredReqCacheMeter[pbsmetrics.CacheHit].Count(), 4) + VerifyMetrics(t, "StoredImpCache.Hit", goEngine.StoredImpCacheMeter[pbsmetrics.CacheHit].Count(), 5) + VerifyMetrics(t, "AccountCache.Hit", goEngine.AccountCacheMeter[pbsmetrics.CacheHit].Count(), 6) } func VerifyMetrics(t *testing.T, name string, actual int64, expected int64) { diff --git a/pbsmetrics/go_metrics.go b/pbsmetrics/go_metrics.go index 621639683ee..aba17d621fc 100644 --- a/pbsmetrics/go_metrics.go +++ b/pbsmetrics/go_metrics.go @@ -27,8 +27,12 @@ type Metrics struct { RequestsQueueTimer map[RequestType]map[bool]metrics.Timer PrebidCacheRequestTimerSuccess metrics.Timer PrebidCacheRequestTimerError metrics.Timer + StoredDataFetchTimer map[StoredDataType]map[StoredDataFetchType]metrics.Timer + StoredDataErrorMeter map[StoredDataType]map[StoredDataError]metrics.Meter StoredReqCacheMeter map[CacheResult]metrics.Meter StoredImpCacheMeter map[CacheResult]metrics.Meter + AccountCacheMeter map[CacheResult]metrics.Meter + DNSLookupTimer metrics.Timer // Metrics for OpenRTB requests specifically. So we can track what % of RequestsMeter are OpenRTB // and know when legacy requests have been abandoned. @@ -48,9 +52,17 @@ type Metrics struct { ImpsTypeAudio metrics.Meter ImpsTypeNative metrics.Meter + // Notification timeout metrics TimeoutNotificationSuccess metrics.Meter TimeoutNotificationFailure metrics.Meter + // TCF adaption metrics + PrivacyCCPARequest metrics.Meter + PrivacyCCPARequestOptOut metrics.Meter + PrivacyCOPPARequest metrics.Meter + PrivacyLMTRequest metrics.Meter + PrivacyTCFRequestVersion map[TCFVersionValue]metrics.Meter + AdapterMetrics map[openrtb_ext.BidderName]*AdapterMetrics // Don't export accountMetrics because we need helper functions here to insure its properly populated dynamically accountMetrics map[string]*accountMetrics @@ -73,6 +85,9 @@ type AdapterMetrics struct { BidsReceivedMeter metrics.Meter PanicMeter metrics.Meter MarkupMetrics map[openrtb_ext.BidType]*MarkupDeliveryMetrics + ConnCreated metrics.Counter + ConnReused metrics.Counter + ConnWaitTime metrics.Timer } type MarkupDeliveryMetrics struct { @@ -98,7 +113,7 @@ const unknownBidder openrtb_ext.BidderName = "unknown" // rather than loading legacy metrics that never get filled. // This will also eventually let us configure metrics, such as setting a limited set of metrics // for a production instance, and then expanding again when we need more debugging. -func NewBlankMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderName, disableMetrics config.DisabledMetrics) *Metrics { +func NewBlankMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderName, disabledMetrics config.DisabledMetrics) *Metrics { blankMeter := &metrics.NilMeter{} blankTimer := &metrics.NilTimer{} @@ -115,11 +130,15 @@ func NewBlankMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderNa SafariRequestMeter: blankMeter, SafariNoCookieMeter: blankMeter, RequestTimer: blankTimer, + DNSLookupTimer: blankTimer, RequestsQueueTimer: make(map[RequestType]map[bool]metrics.Timer), PrebidCacheRequestTimerSuccess: blankTimer, PrebidCacheRequestTimerError: blankTimer, + StoredDataFetchTimer: make(map[StoredDataType]map[StoredDataFetchType]metrics.Timer), + StoredDataErrorMeter: make(map[StoredDataType]map[StoredDataError]metrics.Meter), StoredReqCacheMeter: make(map[CacheResult]metrics.Meter), StoredImpCacheMeter: make(map[CacheResult]metrics.Meter), + AccountCacheMeter: make(map[CacheResult]metrics.Meter), AmpNoCookieMeter: blankMeter, CookieSyncMeter: blankMeter, CookieSyncGen: make(map[openrtb_ext.BidderName]metrics.Meter), @@ -137,14 +156,21 @@ func NewBlankMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderNa TimeoutNotificationSuccess: blankMeter, TimeoutNotificationFailure: blankMeter, + PrivacyCCPARequest: blankMeter, + PrivacyCCPARequestOptOut: blankMeter, + PrivacyCOPPARequest: blankMeter, + PrivacyLMTRequest: blankMeter, + PrivacyTCFRequestVersion: make(map[TCFVersionValue]metrics.Meter, len(TCFVersions())), + AdapterMetrics: make(map[openrtb_ext.BidderName]*AdapterMetrics, len(exchanges)), accountMetrics: make(map[string]*accountMetrics), - MetricsDisabled: disableMetrics, + MetricsDisabled: disabledMetrics, exchanges: exchanges, } + for _, a := range exchanges { - newMetrics.AdapterMetrics[a] = makeBlankAdapterMetrics() + newMetrics.AdapterMetrics[a] = makeBlankAdapterMetrics(newMetrics.MetricsDisabled) } for _, t := range RequestTypes() { @@ -154,6 +180,27 @@ func NewBlankMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderNa } } + for _, c := range CacheResults() { + newMetrics.StoredReqCacheMeter[c] = blankMeter + newMetrics.StoredImpCacheMeter[c] = blankMeter + newMetrics.AccountCacheMeter[c] = blankMeter + } + + for _, v := range TCFVersions() { + newMetrics.PrivacyTCFRequestVersion[v] = blankMeter + } + + for _, dt := range StoredDataTypes() { + newMetrics.StoredDataFetchTimer[dt] = make(map[StoredDataFetchType]metrics.Timer) + newMetrics.StoredDataErrorMeter[dt] = make(map[StoredDataError]metrics.Meter) + for _, ft := range StoredDataFetchTypes() { + newMetrics.StoredDataFetchTimer[dt][ft] = blankTimer + } + for _, e := range StoredDataErrors() { + newMetrics.StoredDataErrorMeter[dt][e] = blankMeter + } + } + //to minimize memory usage, queuedTimeout metric is now supported for video endpoint only //boolean value represents 2 general request statuses: accepted and rejected newMetrics.RequestsQueueTimer["video"] = make(map[bool]metrics.Timer) @@ -185,9 +232,21 @@ func NewMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderName, d newMetrics.AppRequestMeter = metrics.GetOrRegisterMeter("app_requests", registry) newMetrics.SafariNoCookieMeter = metrics.GetOrRegisterMeter("safari_no_cookie_requests", registry) newMetrics.RequestTimer = metrics.GetOrRegisterTimer("request_time", registry) + newMetrics.DNSLookupTimer = metrics.GetOrRegisterTimer("dns_lookup_time", registry) newMetrics.PrebidCacheRequestTimerSuccess = metrics.GetOrRegisterTimer("prebid_cache_request_time.ok", registry) newMetrics.PrebidCacheRequestTimerError = metrics.GetOrRegisterTimer("prebid_cache_request_time.err", registry) + for _, dt := range StoredDataTypes() { + for _, ft := range StoredDataFetchTypes() { + timerName := fmt.Sprintf("stored_%s_fetch_time.%s", string(dt), string(ft)) + newMetrics.StoredDataFetchTimer[dt][ft] = metrics.GetOrRegisterTimer(timerName, registry) + } + for _, e := range StoredDataErrors() { + meterName := fmt.Sprintf("stored_%s_error.%s", string(dt), string(e)) + newMetrics.StoredDataErrorMeter[dt][e] = metrics.GetOrRegisterMeter(meterName, registry) + } + } + newMetrics.AmpNoCookieMeter = metrics.GetOrRegisterMeter("amp_no_cookie_requests", registry) newMetrics.CookieSyncMeter = metrics.GetOrRegisterMeter("cookie_sync_requests", registry) newMetrics.userSyncBadRequest = metrics.GetOrRegisterMeter("usersync.bad_requests", registry) @@ -208,6 +267,7 @@ func NewMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderName, d for _, cacheRes := range CacheResults() { newMetrics.StoredReqCacheMeter[cacheRes] = metrics.GetOrRegisterMeter(fmt.Sprintf("stored_request_cache_%s", string(cacheRes)), registry) newMetrics.StoredImpCacheMeter[cacheRes] = metrics.GetOrRegisterMeter(fmt.Sprintf("stored_imp_cache_%s", string(cacheRes)), registry) + newMetrics.AccountCacheMeter[cacheRes] = metrics.GetOrRegisterMeter(fmt.Sprintf("account_cache_%s", string(cacheRes)), registry) } newMetrics.RequestsQueueTimer["video"][true] = metrics.GetOrRegisterTimer("queued_requests.video.accepted", registry) @@ -218,11 +278,20 @@ func NewMetrics(registry metrics.Registry, exchanges []openrtb_ext.BidderName, d newMetrics.TimeoutNotificationSuccess = metrics.GetOrRegisterMeter("timeout_notification.ok", registry) newMetrics.TimeoutNotificationFailure = metrics.GetOrRegisterMeter("timeout_notification.failed", registry) + + newMetrics.PrivacyCCPARequest = metrics.GetOrRegisterMeter("privacy.request.ccpa.specified", registry) + newMetrics.PrivacyCCPARequestOptOut = metrics.GetOrRegisterMeter("privacy.request.ccpa.opt-out", registry) + newMetrics.PrivacyCOPPARequest = metrics.GetOrRegisterMeter("privacy.request.coppa", registry) + newMetrics.PrivacyLMTRequest = metrics.GetOrRegisterMeter("privacy.request.lmt", registry) + for _, version := range TCFVersions() { + newMetrics.PrivacyTCFRequestVersion[version] = metrics.GetOrRegisterMeter(fmt.Sprintf("privacy.request.tcf.%s", string(version)), registry) + } + return newMetrics } // Part of setting up blank metrics, the adapter metrics. -func makeBlankAdapterMetrics() *AdapterMetrics { +func makeBlankAdapterMetrics(disabledMetrics config.DisabledMetrics) *AdapterMetrics { blankMeter := &metrics.NilMeter{} newAdapter := &AdapterMetrics{ NoCookieMeter: blankMeter, @@ -235,6 +304,11 @@ func makeBlankAdapterMetrics() *AdapterMetrics { PanicMeter: blankMeter, MarkupMetrics: makeBlankBidMarkupMetrics(), } + if !disabledMetrics.AdapterConnectionMetrics { + newAdapter.ConnCreated = metrics.NilCounter{} + newAdapter.ConnReused = metrics.NilCounter{} + newAdapter.ConnWaitTime = &metrics.NilTimer{} + } for _, err := range AdapterErrors() { newAdapter.ErrorMeters[err] = blankMeter } @@ -269,6 +343,9 @@ func registerAdapterMetrics(registry metrics.Registry, adapterOrAccount string, openrtb_ext.BidTypeAudio: makeDeliveryMetrics(registry, adapterOrAccount+"."+exchange, openrtb_ext.BidTypeAudio), openrtb_ext.BidTypeNative: makeDeliveryMetrics(registry, adapterOrAccount+"."+exchange, openrtb_ext.BidTypeNative), } + am.ConnCreated = metrics.GetOrRegisterCounter(fmt.Sprintf("%[1]s.%[2]s.connections_created", adapterOrAccount, exchange), registry) + am.ConnReused = metrics.GetOrRegisterCounter(fmt.Sprintf("%[1]s.%[2]s.connections_reused", adapterOrAccount, exchange), registry) + am.ConnWaitTime = metrics.GetOrRegisterTimer(fmt.Sprintf("%[1]s.%[2]s.connection_wait_time", adapterOrAccount, exchange), registry) for err := range am.ErrorMeters { am.ErrorMeters[err] = metrics.GetOrRegisterMeter(fmt.Sprintf("%s.%s.requests.%s", adapterOrAccount, exchange, err), registry) } @@ -315,7 +392,7 @@ func (me *Metrics) getAccountMetrics(id string) *accountMetrics { am.adapterMetrics = make(map[openrtb_ext.BidderName]*AdapterMetrics, len(me.exchanges)) if !me.MetricsDisabled.AccountAdapterDetails { for _, a := range me.exchanges { - am.adapterMetrics[a] = makeBlankAdapterMetrics() + am.adapterMetrics[a] = makeBlankAdapterMetrics(me.MetricsDisabled) registerAdapterMetrics(me.MetricsRegistry, fmt.Sprintf("account.%s", id), string(a), am.adapterMetrics[a]) } } @@ -397,6 +474,16 @@ func (me *Metrics) RecordRequestTime(labels Labels, length time.Duration) { } } +// RecordStoredDataFetchTime implements a part of the MetricsEngine interface +func (me *Metrics) RecordStoredDataFetchTime(labels StoredDataLabels, length time.Duration) { + me.StoredDataFetchTimer[labels.DataType][labels.DataFetchType].Update(length) +} + +// RecordStoredDataError implements a part of the MetricsEngine interface +func (me *Metrics) RecordStoredDataError(labels StoredDataLabels) { + me.StoredDataErrorMeter[labels.DataType][labels.Error].Mark(1) +} + // RecordAdapterPanic implements a part of the MetricsEngine interface func (me *Metrics) RecordAdapterPanic(labels AdapterLabels) { am, ok := me.AdapterMetrics[labels.Adapter] @@ -439,6 +526,34 @@ func (me *Metrics) RecordAdapterRequest(labels AdapterLabels) { } } +// Keeps track of created and reused connections to adapter bidders and the time from the +// connection request, to the connection creation, or reuse from the pool across all engines +func (me *Metrics) RecordAdapterConnections(adapterName openrtb_ext.BidderName, + connWasReused bool, + connWaitTime time.Duration) { + + if me.MetricsDisabled.AdapterConnectionMetrics { + return + } + + am, ok := me.AdapterMetrics[adapterName] + if !ok { + glog.Errorf("Trying to log adapter connection metrics for %s: adapter not found", string(adapterName)) + return + } + + if connWasReused { + am.ConnReused.Inc(1) + } else { + am.ConnCreated.Inc(1) + } + am.ConnWaitTime.Update(connWaitTime) +} + +func (me *Metrics) RecordDNSTime(dnsLookupTime time.Duration) { + me.DNSLookupTimer.Update(dnsLookupTime) +} + // RecordAdapterBidReceived implements a part of the MetricsEngine interface. // This tracks how many bids from each Bidder use `adm` vs. `nurl. func (me *Metrics) RecordAdapterBidReceived(labels AdapterLabels, bidType openrtb_ext.BidType, hasAdm bool) { @@ -536,6 +651,12 @@ func (me *Metrics) RecordStoredImpCacheResult(cacheResult CacheResult, inc int) me.StoredImpCacheMeter[cacheResult].Mark(int64(inc)) } +// RecordAccountCacheResult implements a part of the MetricsEngine interface. Records the +// cache hits and misses when looking up accounts. +func (me *Metrics) RecordAccountCacheResult(cacheResult CacheResult, inc int) { + me.AccountCacheMeter[cacheResult].Mark(int64(inc)) +} + // RecordPrebidCacheRequestTime implements a part of the MetricsEngine interface. Records the // amount of time taken to store the auction result in Prebid Cache. func (me *Metrics) RecordPrebidCacheRequestTime(success bool, length time.Duration) { @@ -562,6 +683,32 @@ func (me *Metrics) RecordTimeoutNotice(success bool) { return } +func (me *Metrics) RecordRequestPrivacy(privacy PrivacyLabels) { + if privacy.CCPAProvided { + me.PrivacyCCPARequest.Mark(1) + if privacy.CCPAEnforced { + me.PrivacyCCPARequestOptOut.Mark(1) + } + } + + if privacy.COPPAEnforced { + me.PrivacyCOPPARequest.Mark(1) + } + + if privacy.GDPREnforced { + if metric, ok := me.PrivacyTCFRequestVersion[privacy.GDPRTCFVersion]; ok { + metric.Mark(1) + } else { + me.PrivacyTCFRequestVersion[TCFVersionErr].Mark(1) + } + } + + if privacy.LMTEnforced { + me.PrivacyLMTRequest.Mark(1) + } + return +} + // RecordAdapterDuplicateBidID as noop func (me *Metrics) RecordAdapterDuplicateBidID(adaptor string, collisions int) { } diff --git a/pbsmetrics/go_metrics_test.go b/pbsmetrics/go_metrics_test.go index d888385da16..f55e5c9cecc 100644 --- a/pbsmetrics/go_metrics_test.go +++ b/pbsmetrics/go_metrics_test.go @@ -2,6 +2,7 @@ package pbsmetrics import ( "testing" + "time" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" @@ -56,6 +57,14 @@ func TestNewMetrics(t *testing.T) { ensureContains(t, registry, "timeout_notification.ok", m.TimeoutNotificationSuccess) ensureContains(t, registry, "timeout_notification.failed", m.TimeoutNotificationFailure) + + ensureContains(t, registry, "privacy.request.ccpa.specified", m.PrivacyCCPARequest) + ensureContains(t, registry, "privacy.request.ccpa.opt-out", m.PrivacyCCPARequestOptOut) + ensureContains(t, registry, "privacy.request.coppa", m.PrivacyCOPPARequest) + ensureContains(t, registry, "privacy.request.lmt", m.PrivacyLMTRequest) + ensureContains(t, registry, "privacy.request.tcf.v1", m.PrivacyTCFRequestVersion[TCFVersionV1]) + ensureContains(t, registry, "privacy.request.tcf.v2", m.PrivacyTCFRequestVersion[TCFVersionV2]) + ensureContains(t, registry, "privacy.request.tcf.err", m.PrivacyTCFRequestVersion[TCFVersionErr]) } func TestRecordBidType(t *testing.T) { @@ -107,6 +116,10 @@ func ensureContainsAdapterMetrics(t *testing.T, registry metrics.Registry, name ensureContains(t, registry, name+".request_time", adapterMetrics.RequestTimer) ensureContains(t, registry, name+".prices", adapterMetrics.PriceHistogram) ensureContainsBidTypeMetrics(t, registry, name, adapterMetrics.MarkupMetrics) + + ensureContains(t, registry, name+".connections_created", adapterMetrics.ConnCreated) + ensureContains(t, registry, name+".connections_reused", adapterMetrics.ConnReused) + ensureContains(t, registry, name+".connection_wait_time", adapterMetrics.ConnWaitTime) } func TestRecordBidTypeDisabledConfig(t *testing.T) { @@ -171,6 +184,140 @@ func TestRecordBidTypeDisabledConfig(t *testing.T) { } } +func TestRecordDNSTime(t *testing.T) { + testCases := []struct { + description string + inDnsLookupDuration time.Duration + outExpDuration time.Duration + }{ + { + description: "Five second DNS lookup time", + inDnsLookupDuration: time.Second * 5, + outExpDuration: time.Second * 5, + }, + { + description: "Zero DNS lookup time", + inDnsLookupDuration: time.Duration(0), + outExpDuration: time.Duration(0), + }, + } + for _, test := range testCases { + registry := metrics.NewRegistry() + m := NewMetrics(registry, []openrtb_ext.BidderName{openrtb_ext.BidderAppnexus}, config.DisabledMetrics{AccountAdapterDetails: true}) + + m.RecordDNSTime(test.inDnsLookupDuration) + + assert.Equal(t, test.outExpDuration.Nanoseconds(), m.DNSLookupTimer.Sum(), test.description) + } +} + +func TestRecordAdapterConnections(t *testing.T) { + var fakeBidder openrtb_ext.BidderName = "fooAdvertising" + + type testIn struct { + adapterName openrtb_ext.BidderName + connWasReused bool + connWait time.Duration + connMetricsDisabled bool + } + + type testOut struct { + expectedConnReusedCount int64 + expectedConnCreatedCount int64 + expectedConnWaitTime time.Duration + } + + testCases := []struct { + description string + in testIn + out testOut + }{ + { + description: "Successful, new connection created, has connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: false, + connWait: time.Second * 5, + connMetricsDisabled: false, + }, + out: testOut{ + expectedConnReusedCount: 0, + expectedConnCreatedCount: 1, + expectedConnWaitTime: time.Second * 5, + }, + }, + { + description: "Successful, new connection created, has connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: false, + connWait: time.Second * 4, + connMetricsDisabled: false, + }, + out: testOut{ + expectedConnCreatedCount: 1, + expectedConnWaitTime: time.Second * 4, + }, + }, + { + description: "Successful, was reused, no connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: true, + connMetricsDisabled: false, + }, + out: testOut{ + expectedConnReusedCount: 1, + expectedConnWaitTime: 0, + }, + }, + { + description: "Successful, was reused, has connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: true, + connWait: time.Second * 5, + connMetricsDisabled: false, + }, + out: testOut{ + expectedConnReusedCount: 1, + expectedConnWaitTime: time.Second * 5, + }, + }, + { + description: "Fake bidder, nothing gets updated", + in: testIn{ + adapterName: fakeBidder, + connWasReused: false, + connWait: 0, + connMetricsDisabled: false, + }, + out: testOut{}, + }, + { + description: "Adapter connection metrics are disabled, nothing gets updated", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: false, + connWait: time.Second * 5, + connMetricsDisabled: true, + }, + out: testOut{}, + }, + } + + for i, test := range testCases { + registry := metrics.NewRegistry() + m := NewMetrics(registry, []openrtb_ext.BidderName{openrtb_ext.BidderAppnexus}, config.DisabledMetrics{AdapterConnectionMetrics: test.in.connMetricsDisabled}) + + m.RecordAdapterConnections(test.in.adapterName, test.in.connWasReused, test.in.connWait) + + assert.Equal(t, test.out.expectedConnReusedCount, m.AdapterMetrics[openrtb_ext.BidderAppnexus].ConnReused.Count(), "Test [%d] incorrect number of reused connections to adapter", i) + assert.Equal(t, test.out.expectedConnCreatedCount, m.AdapterMetrics[openrtb_ext.BidderAppnexus].ConnCreated.Count(), "Test [%d] incorrect number of new connections to adapter created", i) + assert.Equal(t, test.out.expectedConnWaitTime.Nanoseconds(), m.AdapterMetrics[openrtb_ext.BidderAppnexus].ConnWaitTime.Sum(), "Test [%d] incorrect wait time in connection to adapter", i) + } +} + func TestNewMetricsWithDisabledConfig(t *testing.T) { registry := metrics.NewRegistry() m := NewMetrics(registry, []openrtb_ext.BidderName{openrtb_ext.BidderAppnexus, openrtb_ext.BidderRubicon}, config.DisabledMetrics{AccountAdapterDetails: true}) @@ -198,6 +345,206 @@ func TestRecordPrebidCacheRequestTimeWithNotSuccess(t *testing.T) { assert.Equal(t, m.PrebidCacheRequestTimerError.Count(), int64(1)) } +func TestRecordStoredDataFetchTime(t *testing.T) { + tests := []struct { + description string + dataType StoredDataType + fetchType StoredDataFetchType + }{ + { + description: "Update stored_account_fetch_time.all timer", + dataType: AccountDataType, + fetchType: FetchAll, + }, + { + description: "Update stored_amp_fetch_time.all timer", + dataType: AMPDataType, + fetchType: FetchAll, + }, + { + description: "Update stored_category_fetch_time.all timer", + dataType: CategoryDataType, + fetchType: FetchAll, + }, + { + description: "Update stored_request_fetch_time.all timer", + dataType: RequestDataType, + fetchType: FetchAll, + }, + { + description: "Update stored_video_fetch_time.all timer", + dataType: VideoDataType, + fetchType: FetchAll, + }, + { + description: "Update stored_account_fetch_time.delta timer", + dataType: AccountDataType, + fetchType: FetchDelta, + }, + { + description: "Update stored_amp_fetch_time.delta timer", + dataType: AMPDataType, + fetchType: FetchDelta, + }, + { + description: "Update stored_category_fetch_time.delta timer", + dataType: CategoryDataType, + fetchType: FetchDelta, + }, + { + description: "Update stored_request_fetch_time.delta timer", + dataType: RequestDataType, + fetchType: FetchDelta, + }, + { + description: "Update stored_video_fetch_time.delta timer", + dataType: VideoDataType, + fetchType: FetchDelta, + }, + } + + for _, tt := range tests { + registry := metrics.NewRegistry() + m := NewMetrics(registry, []openrtb_ext.BidderName{openrtb_ext.BidderAppnexus, openrtb_ext.BidderRubicon}, config.DisabledMetrics{AccountAdapterDetails: true}) + m.RecordStoredDataFetchTime(StoredDataLabels{ + DataType: tt.dataType, + DataFetchType: tt.fetchType, + }, time.Duration(500)) + + actualCount := m.StoredDataFetchTimer[tt.dataType][tt.fetchType].Count() + assert.Equal(t, int64(1), actualCount, tt.description) + + actualDuration := m.StoredDataFetchTimer[tt.dataType][tt.fetchType].Sum() + assert.Equal(t, int64(500), actualDuration, tt.description) + } +} + +func TestRecordStoredDataError(t *testing.T) { + tests := []struct { + description string + dataType StoredDataType + errorType StoredDataError + }{ + { + description: "Increment stored_account_error.network meter", + dataType: AccountDataType, + errorType: StoredDataErrorNetwork, + }, + { + description: "Increment stored_amp_error.network meter", + dataType: AMPDataType, + errorType: StoredDataErrorNetwork, + }, + { + description: "Increment stored_category_error.network meter", + dataType: CategoryDataType, + errorType: StoredDataErrorNetwork, + }, + { + description: "Increment stored_request_error.network meter", + dataType: RequestDataType, + errorType: StoredDataErrorNetwork, + }, + { + description: "Increment stored_video_error.network meter", + dataType: VideoDataType, + errorType: StoredDataErrorNetwork, + }, + { + description: "Increment stored_account_error.undefined meter", + dataType: AccountDataType, + errorType: StoredDataErrorUndefined, + }, + { + description: "Increment stored_amp_error.undefined meter", + dataType: AMPDataType, + errorType: StoredDataErrorUndefined, + }, + { + description: "Increment stored_category_error.undefined meter", + dataType: CategoryDataType, + errorType: StoredDataErrorUndefined, + }, + { + description: "Increment stored_request_error.undefined meter", + dataType: RequestDataType, + errorType: StoredDataErrorUndefined, + }, + { + description: "Increment stored_video_error.undefined meter", + dataType: VideoDataType, + errorType: StoredDataErrorUndefined, + }, + } + + for _, tt := range tests { + registry := metrics.NewRegistry() + m := NewMetrics(registry, []openrtb_ext.BidderName{openrtb_ext.BidderAppnexus, openrtb_ext.BidderRubicon}, config.DisabledMetrics{AccountAdapterDetails: true}) + m.RecordStoredDataError(StoredDataLabels{ + DataType: tt.dataType, + Error: tt.errorType, + }) + + actualCount := m.StoredDataErrorMeter[tt.dataType][tt.errorType].Count() + assert.Equal(t, int64(1), actualCount, tt.description) + } +} + +func TestRecordRequestPrivacy(t *testing.T) { + registry := metrics.NewRegistry() + m := NewMetrics(registry, []openrtb_ext.BidderName{openrtb_ext.BidderAppnexus, openrtb_ext.BidderRubicon}, config.DisabledMetrics{AccountAdapterDetails: true}) + + // CCPA + m.RecordRequestPrivacy(PrivacyLabels{ + CCPAEnforced: true, + CCPAProvided: true, + }) + m.RecordRequestPrivacy(PrivacyLabels{ + CCPAEnforced: true, + CCPAProvided: false, + }) + m.RecordRequestPrivacy(PrivacyLabels{ + CCPAEnforced: false, + CCPAProvided: true, + }) + + // COPPA + m.RecordRequestPrivacy(PrivacyLabels{ + COPPAEnforced: true, + }) + + // LMT + m.RecordRequestPrivacy(PrivacyLabels{ + LMTEnforced: true, + }) + + // GDPR + m.RecordRequestPrivacy(PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: TCFVersionErr, + }) + m.RecordRequestPrivacy(PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: TCFVersionV1, + }) + m.RecordRequestPrivacy(PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: TCFVersionV2, + }) + m.RecordRequestPrivacy(PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: TCFVersionV1, + }) + + assert.Equal(t, m.PrivacyCCPARequest.Count(), int64(2), "CCPA") + assert.Equal(t, m.PrivacyCCPARequestOptOut.Count(), int64(1), "CCPA Opt Out") + assert.Equal(t, m.PrivacyCOPPARequest.Count(), int64(1), "COPPA") + assert.Equal(t, m.PrivacyLMTRequest.Count(), int64(1), "LMT") + assert.Equal(t, m.PrivacyTCFRequestVersion[TCFVersionErr].Count(), int64(1), "TCF Err") + assert.Equal(t, m.PrivacyTCFRequestVersion[TCFVersionV1].Count(), int64(2), "TCF V1") + assert.Equal(t, m.PrivacyTCFRequestVersion[TCFVersionV2].Count(), int64(1), "TCF V2") +} + func ensureContainsBidTypeMetrics(t *testing.T, registry metrics.Registry, prefix string, mdm map[openrtb_ext.BidType]*MarkupDeliveryMetrics) { ensureContains(t, registry, prefix+".banner.adm_bids_received", mdm[openrtb_ext.BidTypeBanner].AdmMeter) ensureContains(t, registry, prefix+".banner.nurl_bids_received", mdm[openrtb_ext.BidTypeBanner].NurlMeter) diff --git a/pbsmetrics/metrics.go b/pbsmetrics/metrics.go index a00e24bc7a6..b65a4905296 100644 --- a/pbsmetrics/metrics.go +++ b/pbsmetrics/metrics.go @@ -50,6 +50,70 @@ type RequestLabels struct { RequestStatus RequestStatus } +// PrivacyLabels defines metrics describing the result of privacy enforcement. +type PrivacyLabels struct { + CCPAEnforced bool + CCPAProvided bool + COPPAEnforced bool + GDPREnforced bool + GDPRTCFVersion TCFVersionValue + LMTEnforced bool +} + +type StoredDataType string + +const ( + AccountDataType StoredDataType = "account" + AMPDataType StoredDataType = "amp" + CategoryDataType StoredDataType = "category" + RequestDataType StoredDataType = "request" + VideoDataType StoredDataType = "video" +) + +func StoredDataTypes() []StoredDataType { + return []StoredDataType{ + AccountDataType, + AMPDataType, + CategoryDataType, + RequestDataType, + VideoDataType, + } +} + +type StoredDataFetchType string + +const ( + FetchAll StoredDataFetchType = "all" + FetchDelta StoredDataFetchType = "delta" +) + +func StoredDataFetchTypes() []StoredDataFetchType { + return []StoredDataFetchType{ + FetchAll, + FetchDelta, + } +} + +type StoredDataLabels struct { + DataType StoredDataType + DataFetchType StoredDataFetchType + Error StoredDataError +} + +type StoredDataError string + +const ( + StoredDataErrorNetwork StoredDataError = "network" + StoredDataErrorUndefined StoredDataError = "undefined" +) + +func StoredDataErrors() []StoredDataError { + return []StoredDataError{ + StoredDataErrorNetwork, + StoredDataErrorUndefined, + } +} + // Label typecasting. Se below the type definitions for possible values // DemandSource : Demand source enumeration @@ -257,6 +321,35 @@ func RequestActions() []RequestAction { } } +// TCFVersionValue : The possible values for TCF versions +type TCFVersionValue string + +const ( + TCFVersionErr TCFVersionValue = "err" + TCFVersionV1 TCFVersionValue = "v1" + TCFVersionV2 TCFVersionValue = "v2" +) + +// TCFVersions returns the possible values for the TCF version +func TCFVersions() []TCFVersionValue { + return []TCFVersionValue{ + TCFVersionErr, + TCFVersionV1, + TCFVersionV2, + } +} + +// TCFVersionToValue takes an integer TCF version and returns the corresponding TCFVersionValue +func TCFVersionToValue(version int) TCFVersionValue { + switch { + case version == 1: + return TCFVersionV1 + case version == 2: + return TCFVersionV2 + } + return TCFVersionErr +} + // MetricsEngine is a generic interface to record PBS metrics into the desired backend // The first three metrics function fire off once per incoming request, so total metrics // will equal the total number of incoming requests. The remaining 5 fire off per outgoing @@ -271,6 +364,8 @@ type MetricsEngine interface { RecordLegacyImps(labels Labels, numImps int) // RecordImps for the legacy engine RecordRequestTime(labels Labels, length time.Duration) // ignores adapter. only statusOk and statusErr fom status RecordAdapterRequest(labels AdapterLabels) + RecordAdapterConnections(adapterName openrtb_ext.BidderName, connWasReused bool, connWaitTime time.Duration) + RecordDNSTime(dnsLookupTime time.Duration) RecordAdapterPanic(labels AdapterLabels) // This records whether or not a bid of a particular type uses `adm` or `nurl`. // Since the legacy endpoints don't have a bid type, it can only count bids from OpenRTB and AMP. @@ -282,9 +377,14 @@ type MetricsEngine interface { RecordUserIDSet(userLabels UserLabels) // Function should verify bidder values RecordStoredReqCacheResult(cacheResult CacheResult, inc int) RecordStoredImpCacheResult(cacheResult CacheResult, inc int) + RecordAccountCacheResult(cacheResult CacheResult, inc int) + RecordStoredDataFetchTime(labels StoredDataLabels, length time.Duration) + RecordStoredDataError(labels StoredDataLabels) RecordPrebidCacheRequestTime(success bool, length time.Duration) RecordRequestQueueTime(success bool, requestType RequestType, length time.Duration) RecordTimeoutNotice(sucess bool) + RecordRequestPrivacy(privacy PrivacyLabels) + // RecordAdapterDuplicateBidID captures the bid.ID collisions when adaptor // gives the bid response with multiple bids containing same bid.ID RecordAdapterDuplicateBidID(adaptor string, collisions int) diff --git a/pbsmetrics/metrics_mock.go b/pbsmetrics/metrics_mock.go index dac5c88e7ac..8f6710e6339 100644 --- a/pbsmetrics/metrics_mock.go +++ b/pbsmetrics/metrics_mock.go @@ -42,6 +42,16 @@ func (me *MetricsEngineMock) RecordRequestTime(labels Labels, length time.Durati me.Called(labels, length) } +// RecordStoredDataFetchTime mock +func (me *MetricsEngineMock) RecordStoredDataFetchTime(labels StoredDataLabels, length time.Duration) { + me.Called(labels, length) +} + +// RecordStoredDataError mock +func (me *MetricsEngineMock) RecordStoredDataError(labels StoredDataLabels) { + me.Called(labels) +} + // RecordAdapterPanic mock func (me *MetricsEngineMock) RecordAdapterPanic(labels AdapterLabels) { me.Called(labels) @@ -52,6 +62,16 @@ func (me *MetricsEngineMock) RecordAdapterRequest(labels AdapterLabels) { me.Called(labels) } +// RecordAdapterConnections mock +func (me *MetricsEngineMock) RecordAdapterConnections(bidderName openrtb_ext.BidderName, connWasReused bool, connWaitTime time.Duration) { + me.Called(bidderName, connWasReused, connWaitTime) +} + +// RecordDNSTime mock +func (me *MetricsEngineMock) RecordDNSTime(dnsLookupTime time.Duration) { + me.Called(dnsLookupTime) +} + // RecordAdapterBidReceived mock func (me *MetricsEngineMock) RecordAdapterBidReceived(labels AdapterLabels, bidType openrtb_ext.BidType, hasAdm bool) { me.Called(labels, bidType, hasAdm) @@ -92,6 +112,11 @@ func (me *MetricsEngineMock) RecordStoredImpCacheResult(cacheResult CacheResult, me.Called(cacheResult, inc) } +// RecordAccountCacheResult mock +func (me *MetricsEngineMock) RecordAccountCacheResult(cacheResult CacheResult, inc int) { + me.Called(cacheResult, inc) +} + // RecordPrebidCacheRequestTime mock func (me *MetricsEngineMock) RecordPrebidCacheRequestTime(success bool, length time.Duration) { me.Called(success, length) @@ -107,6 +132,11 @@ func (me *MetricsEngineMock) RecordTimeoutNotice(success bool) { me.Called(success) } +// RecordRequestPrivacy mock +func (me *MetricsEngineMock) RecordRequestPrivacy(privacy PrivacyLabels) { + me.Called(privacy) +} + // RecordAdapterDuplicateBidID mock func (me *MetricsEngineMock) RecordAdapterDuplicateBidID(adaptor string, collisions int) { me.Called(adaptor, collisions) diff --git a/pbsmetrics/prometheus/preload.go b/pbsmetrics/prometheus/preload.go index e27451c4bd6..4091d19ea3f 100644 --- a/pbsmetrics/prometheus/preload.go +++ b/pbsmetrics/prometheus/preload.go @@ -7,16 +7,19 @@ import ( func preloadLabelValues(m *Metrics) { var ( - actionValues = actionsAsString() - adapterValues = adaptersAsString() - adapterErrorValues = adapterErrorsAsString() - bidTypeValues = []string{markupDeliveryAdm, markupDeliveryNurl} - boolValues = boolValuesAsString() - cacheResultValues = cacheResultsAsString() - cookieValues = cookieTypesAsString() - connectionErrorValues = []string{connectionAcceptError, connectionCloseError} - requestStatusValues = requestStatusesAsString() - requestTypeValues = requestTypesAsString() + actionValues = actionsAsString() + adapterErrorValues = adapterErrorsAsString() + adapterValues = adaptersAsString() + bidTypeValues = []string{markupDeliveryAdm, markupDeliveryNurl} + boolValues = boolValuesAsString() + cacheResultValues = cacheResultsAsString() + connectionErrorValues = []string{connectionAcceptError, connectionCloseError} + cookieValues = cookieTypesAsString() + requestStatusValues = requestStatusesAsString() + requestTypeValues = requestTypesAsString() + storedDataFetchTypeValues = storedDataFetchTypesAsString() + storedDataErrorValues = storedDataErrorsAsString() + sourceValues = []string{sourceRequest} ) preloadLabelValuesForCounter(m.connectionsError, map[string][]string{ @@ -43,6 +46,46 @@ func preloadLabelValues(m *Metrics) { requestTypeLabel: requestTypeValues, }) + preloadLabelValuesForHistogram(m.storedAccountFetchTimer, map[string][]string{ + storedDataFetchTypeLabel: storedDataFetchTypeValues, + }) + + preloadLabelValuesForHistogram(m.storedAMPFetchTimer, map[string][]string{ + storedDataFetchTypeLabel: storedDataFetchTypeValues, + }) + + preloadLabelValuesForHistogram(m.storedCategoryFetchTimer, map[string][]string{ + storedDataFetchTypeLabel: storedDataFetchTypeValues, + }) + + preloadLabelValuesForHistogram(m.storedRequestFetchTimer, map[string][]string{ + storedDataFetchTypeLabel: storedDataFetchTypeValues, + }) + + preloadLabelValuesForHistogram(m.storedVideoFetchTimer, map[string][]string{ + storedDataFetchTypeLabel: storedDataFetchTypeValues, + }) + + preloadLabelValuesForCounter(m.storedAccountErrors, map[string][]string{ + storedDataErrorLabel: storedDataErrorValues, + }) + + preloadLabelValuesForCounter(m.storedAMPErrors, map[string][]string{ + storedDataErrorLabel: storedDataErrorValues, + }) + + preloadLabelValuesForCounter(m.storedCategoryErrors, map[string][]string{ + storedDataErrorLabel: storedDataErrorValues, + }) + + preloadLabelValuesForCounter(m.storedRequestErrors, map[string][]string{ + storedDataErrorLabel: storedDataErrorValues, + }) + + preloadLabelValuesForCounter(m.storedVideoErrors, map[string][]string{ + storedDataErrorLabel: storedDataErrorValues, + }) + preloadLabelValuesForCounter(m.requestsWithoutCookie, map[string][]string{ requestTypeLabel: requestTypeValues, }) @@ -55,6 +98,10 @@ func preloadLabelValues(m *Metrics) { cacheResultLabel: cacheResultValues, }) + preloadLabelValuesForCounter(m.accountCacheResult, map[string][]string{ + cacheResultLabel: cacheResultValues, + }) + preloadLabelValuesForCounter(m.adapterBids, map[string][]string{ adapterLabel: adapterValues, markupDeliveryLabel: bidTypeValues, @@ -84,6 +131,20 @@ func preloadLabelValues(m *Metrics) { hasBidsLabel: boolValues, }) + if !m.metricsDisabled.AdapterConnectionMetrics { + preloadLabelValuesForCounter(m.adapterCreatedConnections, map[string][]string{ + adapterLabel: adapterValues, + }) + + preloadLabelValuesForCounter(m.adapterReusedConnections, map[string][]string{ + adapterLabel: adapterValues, + }) + + preloadLabelValuesForHistogram(m.adapterConnectionWaitTime, map[string][]string{ + adapterLabel: adapterValues, + }) + } + preloadLabelValuesForHistogram(m.adapterRequestsTimer, map[string][]string{ adapterLabel: adapterValues, }) @@ -99,6 +160,24 @@ func preloadLabelValues(m *Metrics) { requestTypeLabel: {string(pbsmetrics.ReqTypeVideo)}, requestStatusLabel: {requestSuccessLabel, requestRejectLabel}, }) + + preloadLabelValuesForCounter(m.privacyCCPA, map[string][]string{ + sourceLabel: sourceValues, + optOutLabel: boolValues, + }) + + preloadLabelValuesForCounter(m.privacyCOPPA, map[string][]string{ + sourceLabel: sourceValues, + }) + + preloadLabelValuesForCounter(m.privacyLMT, map[string][]string{ + sourceLabel: sourceValues, + }) + + preloadLabelValuesForCounter(m.privacyTCF, map[string][]string{ + sourceLabel: sourceValues, + versionLabel: tcfVersionsAsString(), + }) } func preloadLabelValuesForCounter(counter *prometheus.CounterVec, labelsWithValues map[string][]string) { diff --git a/pbsmetrics/prometheus/prometheus.go b/pbsmetrics/prometheus/prometheus.go index fe621e779d7..54b7810fef4 100644 --- a/pbsmetrics/prometheus/prometheus.go +++ b/pbsmetrics/prometheus/prometheus.go @@ -28,6 +28,23 @@ type Metrics struct { requestsWithoutCookie *prometheus.CounterVec storedImpressionsCacheResult *prometheus.CounterVec storedRequestCacheResult *prometheus.CounterVec + accountCacheResult *prometheus.CounterVec + storedAccountFetchTimer *prometheus.HistogramVec + storedAccountErrors *prometheus.CounterVec + storedAMPFetchTimer *prometheus.HistogramVec + storedAMPErrors *prometheus.CounterVec + storedCategoryFetchTimer *prometheus.HistogramVec + storedCategoryErrors *prometheus.CounterVec + storedRequestFetchTimer *prometheus.HistogramVec + storedRequestErrors *prometheus.CounterVec + storedVideoFetchTimer *prometheus.HistogramVec + storedVideoErrors *prometheus.CounterVec + timeoutNotifications *prometheus.CounterVec + dnsLookupTimer prometheus.Histogram + privacyCCPA *prometheus.CounterVec + privacyCOPPA *prometheus.CounterVec + privacyLMT *prometheus.CounterVec + privacyTCF *prometheus.CounterVec timeout_notifications *prometheus.CounterVec requestsDuplicateBidIDCounter prometheus.Counter // total request having duplicate bid.id for given bidder @@ -40,6 +57,9 @@ type Metrics struct { adapterRequests *prometheus.CounterVec adapterRequestsTimer *prometheus.HistogramVec adapterUserSync *prometheus.CounterVec + adapterReusedConnections *prometheus.CounterVec + adapterCreatedConnections *prometheus.CounterVec + adapterConnectionWaitTime *prometheus.HistogramVec adapterDuplicateBidIDCounter *prometheus.CounterVec adapterVideoBidDuration *prometheus.HistogramVec @@ -59,6 +79,8 @@ type Metrics struct { // podCompExclTimer indicates time taken by compititve exclusion // algorithm to generate final pod response based on bid response and ad pod request podCompExclTimer *prometheus.HistogramVec + + metricsDisabled config.DisabledMetrics } const ( @@ -76,10 +98,12 @@ const ( isNativeLabel = "native" isVideoLabel = "video" markupDeliveryLabel = "delivery" + optOutLabel = "opt_out" privacyBlockedLabel = "privacy_blocked" requestStatusLabel = "request_status" requestTypeLabel = "request_type" successLabel = "success" + versionLabel = "version" ) const ( @@ -110,15 +134,26 @@ const ( podNoOfResponseBids = "no_of_response_bids" ) +const ( + sourceLabel = "source" + sourceRequest = "request" +) + +const ( + storedDataFetchTypeLabel = "stored_data_fetch_type" + storedDataErrorLabel = "stored_data_error" +) + // NewMetrics initializes a new Prometheus metrics instance with preloaded label values. -func NewMetrics(cfg config.PrometheusMetrics) *Metrics { - requestTimeBuckets := []float64{0.05, 0.1, 0.15, 0.20, 0.25, 0.3, 0.4, 0.5, 0.75, 1} +func NewMetrics(cfg config.PrometheusMetrics, disabledMetrics config.DisabledMetrics) *Metrics { + standardTimeBuckets := []float64{0.05, 0.1, 0.15, 0.20, 0.25, 0.3, 0.4, 0.5, 0.75, 1} cacheWriteTimeBuckets := []float64{0.001, 0.002, 0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 1} priceBuckets := []float64{250, 500, 750, 1000, 1500, 2000, 2500, 3000, 3500, 4000} queuedRequestTimeBuckets := []float64{0, 1, 5, 30, 60, 120, 180, 240, 300} metrics := Metrics{} metrics.Registry = prometheus.NewRegistry() + metrics.metricsDisabled = disabledMetrics metrics.connectionsClosed = newCounterWithoutLabels(cfg, metrics.Registry, "connections_closed", @@ -146,7 +181,7 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "impressions_requests_legacy", "Count of requested impressions to Prebid Server using the legacy endpoint.") - metrics.prebidCacheWriteTimer = newHistogram(cfg, metrics.Registry, + metrics.prebidCacheWriteTimer = newHistogramVec(cfg, metrics.Registry, "prebidcache_write_time_seconds", "Seconds to write to Prebid Cache labeled by success or failure. Failure timing is limited by Prebid Server enforced timeouts.", []string{successLabel}, @@ -157,11 +192,11 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "Count of total requests to Prebid Server labeled by type and status.", []string{requestTypeLabel, requestStatusLabel}) - metrics.requestsTimer = newHistogram(cfg, metrics.Registry, + metrics.requestsTimer = newHistogramVec(cfg, metrics.Registry, "request_time_seconds", "Seconds to resolve successful Prebid Server requests labeled by type.", []string{requestTypeLabel}, - requestTimeBuckets) + standardTimeBuckets) metrics.requestsWithoutCookie = newCounter(cfg, metrics.Registry, "requests_without_cookie", @@ -178,11 +213,96 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "Count of stored request cache requests attempts by hits or miss.", []string{cacheResultLabel}) - metrics.timeout_notifications = newCounter(cfg, metrics.Registry, + metrics.accountCacheResult = newCounter(cfg, metrics.Registry, + "account_cache_performance", + "Count of account cache lookups by hits or miss.", + []string{cacheResultLabel}) + + metrics.storedAccountFetchTimer = newHistogramVec(cfg, metrics.Registry, + "stored_account_fetch_time_seconds", + "Seconds to fetch stored accounts labeled by fetch type", + []string{storedDataFetchTypeLabel}, + standardTimeBuckets) + + metrics.storedAccountErrors = newCounter(cfg, metrics.Registry, + "stored_account_errors", + "Count of stored account errors by error type", + []string{storedDataErrorLabel}) + + metrics.storedAMPFetchTimer = newHistogramVec(cfg, metrics.Registry, + "stored_amp_fetch_time_seconds", + "Seconds to fetch stored AMP requests labeled by fetch type", + []string{storedDataFetchTypeLabel}, + standardTimeBuckets) + + metrics.storedAMPErrors = newCounter(cfg, metrics.Registry, + "stored_amp_errors", + "Count of stored AMP errors by error type", + []string{storedDataErrorLabel}) + + metrics.storedCategoryFetchTimer = newHistogramVec(cfg, metrics.Registry, + "stored_category_fetch_time_seconds", + "Seconds to fetch stored categories labeled by fetch type", + []string{storedDataFetchTypeLabel}, + standardTimeBuckets) + + metrics.storedCategoryErrors = newCounter(cfg, metrics.Registry, + "stored_category_errors", + "Count of stored category errors by error type", + []string{storedDataErrorLabel}) + + metrics.storedRequestFetchTimer = newHistogramVec(cfg, metrics.Registry, + "stored_request_fetch_time_seconds", + "Seconds to fetch stored requests labeled by fetch type", + []string{storedDataFetchTypeLabel}, + standardTimeBuckets) + + metrics.storedRequestErrors = newCounter(cfg, metrics.Registry, + "stored_request_errors", + "Count of stored request errors by error type", + []string{storedDataErrorLabel}) + + metrics.storedVideoFetchTimer = newHistogramVec(cfg, metrics.Registry, + "stored_video_fetch_time_seconds", + "Seconds to fetch stored video labeled by fetch type", + []string{storedDataFetchTypeLabel}, + standardTimeBuckets) + + metrics.storedVideoErrors = newCounter(cfg, metrics.Registry, + "stored_video_errors", + "Count of stored video errors by error type", + []string{storedDataErrorLabel}) + + metrics.timeoutNotifications = newCounter(cfg, metrics.Registry, "timeout_notification", "Count of timeout notifications triggered, and if they were successfully sent.", []string{successLabel}) + metrics.dnsLookupTimer = newHistogram(cfg, metrics.Registry, + "dns_lookup_time", + "Seconds to resolve DNS", + standardTimeBuckets) + + metrics.privacyCCPA = newCounter(cfg, metrics.Registry, + "privacy_ccpa", + "Count of total requests to Prebid Server where CCPA was provided by source and opt-out .", + []string{sourceLabel, optOutLabel}) + + metrics.privacyCOPPA = newCounter(cfg, metrics.Registry, + "privacy_coppa", + "Count of total requests to Prebid Server where the COPPA flag was set by source", + []string{sourceLabel}) + + metrics.privacyTCF = newCounter(cfg, metrics.Registry, + "privacy_tcf", + "Count of TCF versions for requests where GDPR was enforced by source and version.", + []string{versionLabel, sourceLabel}) + + metrics.privacyLMT = newCounter(cfg, metrics.Registry, + "privacy_lmt", + "Count of total requests to Prebid Server where the LMT flag was set by source", + []string{sourceLabel}) + metrics.adapterBids = newCounter(cfg, metrics.Registry, "adapter_bids", "Count of bids labeled by adapter and markup delivery type (adm or nurl).", @@ -203,7 +323,7 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "Count of panics labeled by adapter.", []string{adapterLabel}) - metrics.adapterPrices = newHistogram(cfg, metrics.Registry, + metrics.adapterPrices = newHistogramVec(cfg, metrics.Registry, "adapter_prices", "Monetary value of the bids labeled by adapter.", []string{adapterLabel}, @@ -214,11 +334,29 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "Count of requests labeled by adapter, if has a cookie, and if it resulted in bids.", []string{adapterLabel, cookieLabel, hasBidsLabel}) - metrics.adapterRequestsTimer = newHistogram(cfg, metrics.Registry, + if !metrics.metricsDisabled.AdapterConnectionMetrics { + metrics.adapterCreatedConnections = newCounter(cfg, metrics.Registry, + "adapter_connection_created", + "Count that keeps track of new connections when contacting adapter bidder endpoints.", + []string{adapterLabel}) + + metrics.adapterReusedConnections = newCounter(cfg, metrics.Registry, + "adapter_connection_reused", + "Count that keeps track of reused connections when contacting adapter bidder endpoints.", + []string{adapterLabel}) + + metrics.adapterConnectionWaitTime = newHistogramVec(cfg, metrics.Registry, + "adapter_connection_wait", + "Seconds from when the connection was requested until it is either created or reused", + []string{adapterLabel}, + standardTimeBuckets) + } + + metrics.adapterRequestsTimer = newHistogramVec(cfg, metrics.Registry, "adapter_request_time_seconds", "Seconds to resolve each successful request labeled by adapter.", []string{adapterLabel}, - requestTimeBuckets) + standardTimeBuckets) metrics.adapterUserSync = newCounter(cfg, metrics.Registry, "adapter_user_sync", @@ -230,7 +368,7 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "Count of total requests to Prebid Server labeled by account.", []string{accountLabel}) - metrics.requestsQueueTimer = newHistogram(cfg, metrics.Registry, + metrics.requestsQueueTimer = newHistogramVec(cfg, metrics.Registry, "request_queue_time", "Seconds request was waiting in queue", []string{requestTypeLabel, requestStatusLabel}, @@ -246,7 +384,7 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { "Count of number of request where bid collision is detected.") // adpod specific metrics - metrics.podImpGenTimer = newHistogram(cfg, metrics.Registry, + metrics.podImpGenTimer = newHistogramVec(cfg, metrics.Registry, "impr_gen", "Time taken by Ad Pod Impression Generator in seconds", []string{podAlgorithm, podNoOfImpressions}, // 200 µS, 250 µS, 275 µS, 300 µS @@ -254,14 +392,14 @@ func NewMetrics(cfg config.PrometheusMetrics) *Metrics { // 100 µS, 200 µS, 300 µS, 400 µS, 500 µS, 600 µS, []float64{0.000100000, 0.000200000, 0.000300000, 0.000400000, 0.000500000, 0.000600000}) - metrics.podCombGenTimer = newHistogram(cfg, metrics.Registry, + metrics.podCombGenTimer = newHistogramVec(cfg, metrics.Registry, "comb_gen", "Time taken by Ad Pod Combination Generator in seconds", []string{podAlgorithm, podTotalCombinations}, // 200 µS, 250 µS, 275 µS, 300 µS //[]float64{0.000200000, 0.000250000, 0.000275000, 0.000300000}) []float64{0.000100000, 0.000200000, 0.000300000, 0.000400000, 0.000500000, 0.000600000}) - metrics.podCompExclTimer = newHistogram(cfg, metrics.Registry, + metrics.podCompExclTimer = newHistogramVec(cfg, metrics.Registry, "comp_excl", "Time taken by Ad Pod Compititve Exclusion in seconds", []string{podAlgorithm, podNoOfResponseBids}, // 200 µS, 250 µS, 275 µS, 300 µS @@ -301,7 +439,7 @@ func newCounterWithoutLabels(cfg config.PrometheusMetrics, registry *prometheus. return counter } -func newHistogram(cfg config.PrometheusMetrics, registry *prometheus.Registry, name, help string, labels []string, buckets []float64) *prometheus.HistogramVec { +func newHistogramVec(cfg config.PrometheusMetrics, registry *prometheus.Registry, name, help string, labels []string, buckets []float64) *prometheus.HistogramVec { opts := prometheus.HistogramOpts{ Namespace: cfg.Namespace, Subsystem: cfg.Subsystem, @@ -314,6 +452,19 @@ func newHistogram(cfg config.PrometheusMetrics, registry *prometheus.Registry, n return histogram } +func newHistogram(cfg config.PrometheusMetrics, registry *prometheus.Registry, name, help string, buckets []float64) prometheus.Histogram { + opts := prometheus.HistogramOpts{ + Namespace: cfg.Namespace, + Subsystem: cfg.Subsystem, + Name: name, + Help: help, + Buckets: buckets, + } + histogram := prometheus.NewHistogram(opts) + registry.MustRegister(histogram) + return histogram +} + func (m *Metrics) RecordConnectionAccept(success bool) { if success { m.connectionsOpened.Inc() @@ -374,6 +525,56 @@ func (m *Metrics) RecordRequestTime(labels pbsmetrics.Labels, length time.Durati } } +func (m *Metrics) RecordStoredDataFetchTime(labels pbsmetrics.StoredDataLabels, length time.Duration) { + switch labels.DataType { + case pbsmetrics.AccountDataType: + m.storedAccountFetchTimer.With(prometheus.Labels{ + storedDataFetchTypeLabel: string(labels.DataFetchType), + }).Observe(length.Seconds()) + case pbsmetrics.AMPDataType: + m.storedAMPFetchTimer.With(prometheus.Labels{ + storedDataFetchTypeLabel: string(labels.DataFetchType), + }).Observe(length.Seconds()) + case pbsmetrics.CategoryDataType: + m.storedCategoryFetchTimer.With(prometheus.Labels{ + storedDataFetchTypeLabel: string(labels.DataFetchType), + }).Observe(length.Seconds()) + case pbsmetrics.RequestDataType: + m.storedRequestFetchTimer.With(prometheus.Labels{ + storedDataFetchTypeLabel: string(labels.DataFetchType), + }).Observe(length.Seconds()) + case pbsmetrics.VideoDataType: + m.storedVideoFetchTimer.With(prometheus.Labels{ + storedDataFetchTypeLabel: string(labels.DataFetchType), + }).Observe(length.Seconds()) + } +} + +func (m *Metrics) RecordStoredDataError(labels pbsmetrics.StoredDataLabels) { + switch labels.DataType { + case pbsmetrics.AccountDataType: + m.storedAccountErrors.With(prometheus.Labels{ + storedDataErrorLabel: string(labels.Error), + }).Inc() + case pbsmetrics.AMPDataType: + m.storedAMPErrors.With(prometheus.Labels{ + storedDataErrorLabel: string(labels.Error), + }).Inc() + case pbsmetrics.CategoryDataType: + m.storedCategoryErrors.With(prometheus.Labels{ + storedDataErrorLabel: string(labels.Error), + }).Inc() + case pbsmetrics.RequestDataType: + m.storedRequestErrors.With(prometheus.Labels{ + storedDataErrorLabel: string(labels.Error), + }).Inc() + case pbsmetrics.VideoDataType: + m.storedVideoErrors.With(prometheus.Labels{ + storedDataErrorLabel: string(labels.Error), + }).Inc() + } +} + func (m *Metrics) RecordAdapterRequest(labels pbsmetrics.AdapterLabels) { m.adapterRequests.With(prometheus.Labels{ adapterLabel: string(labels.Adapter), @@ -389,6 +590,32 @@ func (m *Metrics) RecordAdapterRequest(labels pbsmetrics.AdapterLabels) { } } +// Keeps track of created and reused connections to adapter bidders and the time from the +// connection request, to the connection creation, or reuse from the pool across all engines +func (m *Metrics) RecordAdapterConnections(adapterName openrtb_ext.BidderName, connWasReused bool, connWaitTime time.Duration) { + if m.metricsDisabled.AdapterConnectionMetrics { + return + } + + if connWasReused { + m.adapterReusedConnections.With(prometheus.Labels{ + adapterLabel: string(adapterName), + }).Inc() + } else { + m.adapterCreatedConnections.With(prometheus.Labels{ + adapterLabel: string(adapterName), + }).Inc() + } + + m.adapterConnectionWaitTime.With(prometheus.Labels{ + adapterLabel: string(adapterName), + }).Observe(connWaitTime.Seconds()) +} + +func (m *Metrics) RecordDNSTime(dnsLookupTime time.Duration) { + m.dnsLookupTimer.Observe(dnsLookupTime.Seconds()) +} + func (m *Metrics) RecordAdapterPanic(labels pbsmetrics.AdapterLabels) { m.adapterPanics.With(prometheus.Labels{ adapterLabel: string(labels.Adapter), @@ -454,6 +681,12 @@ func (m *Metrics) RecordStoredImpCacheResult(cacheResult pbsmetrics.CacheResult, }).Add(float64(inc)) } +func (m *Metrics) RecordAccountCacheResult(cacheResult pbsmetrics.CacheResult, inc int) { + m.accountCacheResult.With(prometheus.Labels{ + cacheResultLabel: string(cacheResult), + }).Add(float64(inc)) +} + func (m *Metrics) RecordPrebidCacheRequestTime(success bool, length time.Duration) { m.prebidCacheWriteTimer.With(prometheus.Labels{ successLabel: strconv.FormatBool(success), @@ -473,16 +706,44 @@ func (m *Metrics) RecordRequestQueueTime(success bool, requestType pbsmetrics.Re func (m *Metrics) RecordTimeoutNotice(success bool) { if success { - m.timeout_notifications.With(prometheus.Labels{ + m.timeoutNotifications.With(prometheus.Labels{ successLabel: requestSuccessful, }).Inc() } else { - m.timeout_notifications.With(prometheus.Labels{ + m.timeoutNotifications.With(prometheus.Labels{ successLabel: requestFailed, }).Inc() } } +func (m *Metrics) RecordRequestPrivacy(privacy pbsmetrics.PrivacyLabels) { + if privacy.CCPAProvided { + m.privacyCCPA.With(prometheus.Labels{ + sourceLabel: sourceRequest, + optOutLabel: strconv.FormatBool(privacy.CCPAEnforced), + }).Inc() + } + + if privacy.COPPAEnforced { + m.privacyCOPPA.With(prometheus.Labels{ + sourceLabel: sourceRequest, + }).Inc() + } + + if privacy.GDPREnforced { + m.privacyTCF.With(prometheus.Labels{ + versionLabel: string(privacy.GDPRTCFVersion), + sourceLabel: sourceRequest, + }).Inc() + } + + if privacy.LMTEnforced { + m.privacyLMT.With(prometheus.Labels{ + sourceLabel: sourceRequest, + }).Inc() + } +} + // RecordAdapterDuplicateBidID captures the bid.ID collisions when adaptor // gives the bid response with multiple bids containing same bid.ID // ensure collisions value is greater than 1. This function will not give any error diff --git a/pbsmetrics/prometheus/prometheus_test.go b/pbsmetrics/prometheus/prometheus_test.go index d3e42ff636c..9f6a91f9384 100644 --- a/pbsmetrics/prometheus/prometheus_test.go +++ b/pbsmetrics/prometheus/prometheus_test.go @@ -1,7 +1,10 @@ package prometheusmetrics import ( + "fmt" + "strconv" + "testing" "time" @@ -18,7 +21,7 @@ func createMetricsForTesting() *Metrics { Port: 8080, Namespace: "prebid", Subsystem: "server", - }) + }, config.DisabledMetrics{}) } func TestMetricCountGatekeeping(t *testing.T) { @@ -62,7 +65,7 @@ func TestMetricCountGatekeeping(t *testing.T) { // Verify Per-Adapter Cardinality // - This assertion provides a warning for newly added adapter metrics. Threre are 40+ adapters which makes the // cost of new per-adapter metrics rather expensive. Thought should be given when adding new per-adapter metrics. - assert.True(t, perAdapterCardinalityCount <= 22, "Per-Adapter Cardinality") + assert.True(t, perAdapterCardinalityCount <= 27, "Per-Adapter Cardinality count equals %d \n", perAdapterCardinalityCount) } func TestConnectionMetrics(t *testing.T) { @@ -407,6 +410,193 @@ func TestRequestTimeMetric(t *testing.T) { } } +func TestRecordStoredDataFetchTime(t *testing.T) { + tests := []struct { + description string + dataType pbsmetrics.StoredDataType + fetchType pbsmetrics.StoredDataFetchType + }{ + { + description: "Update stored account histogram with all label", + dataType: pbsmetrics.AccountDataType, + fetchType: pbsmetrics.FetchAll, + }, + { + description: "Update stored AMP histogram with all label", + dataType: pbsmetrics.AMPDataType, + fetchType: pbsmetrics.FetchAll, + }, + { + description: "Update stored category histogram with all label", + dataType: pbsmetrics.CategoryDataType, + fetchType: pbsmetrics.FetchAll, + }, + { + description: "Update stored request histogram with all label", + dataType: pbsmetrics.RequestDataType, + fetchType: pbsmetrics.FetchAll, + }, + { + description: "Update stored video histogram with all label", + dataType: pbsmetrics.VideoDataType, + fetchType: pbsmetrics.FetchAll, + }, + { + description: "Update stored account histogram with delta label", + dataType: pbsmetrics.AccountDataType, + fetchType: pbsmetrics.FetchDelta, + }, + { + description: "Update stored AMP histogram with delta label", + dataType: pbsmetrics.AMPDataType, + fetchType: pbsmetrics.FetchDelta, + }, + { + description: "Update stored category histogram with delta label", + dataType: pbsmetrics.CategoryDataType, + fetchType: pbsmetrics.FetchDelta, + }, + { + description: "Update stored request histogram with delta label", + dataType: pbsmetrics.RequestDataType, + fetchType: pbsmetrics.FetchDelta, + }, + { + description: "Update stored video histogram with delta label", + dataType: pbsmetrics.VideoDataType, + fetchType: pbsmetrics.FetchDelta, + }, + } + + for _, tt := range tests { + m := createMetricsForTesting() + + fetchTime := time.Duration(0.5 * float64(time.Second)) + m.RecordStoredDataFetchTime(pbsmetrics.StoredDataLabels{ + DataType: tt.dataType, + DataFetchType: tt.fetchType, + }, fetchTime) + + var metricsTimer *prometheus.HistogramVec + switch tt.dataType { + case pbsmetrics.AccountDataType: + metricsTimer = m.storedAccountFetchTimer + case pbsmetrics.AMPDataType: + metricsTimer = m.storedAMPFetchTimer + case pbsmetrics.CategoryDataType: + metricsTimer = m.storedCategoryFetchTimer + case pbsmetrics.RequestDataType: + metricsTimer = m.storedRequestFetchTimer + case pbsmetrics.VideoDataType: + metricsTimer = m.storedVideoFetchTimer + } + + result := getHistogramFromHistogramVec( + metricsTimer, + storedDataFetchTypeLabel, + string(tt.fetchType)) + assertHistogram(t, tt.description, result, 1, 0.5) + } +} + +func TestRecordStoredDataError(t *testing.T) { + tests := []struct { + description string + dataType pbsmetrics.StoredDataType + errorType pbsmetrics.StoredDataError + metricName string + }{ + { + description: "Update stored_account_errors counter with network label", + dataType: pbsmetrics.AccountDataType, + errorType: pbsmetrics.StoredDataErrorNetwork, + metricName: "stored_account_errors", + }, + { + description: "Update stored_amp_errors counter with network label", + dataType: pbsmetrics.AMPDataType, + errorType: pbsmetrics.StoredDataErrorNetwork, + metricName: "stored_amp_errors", + }, + { + description: "Update stored_category_errors counter with network label", + dataType: pbsmetrics.CategoryDataType, + errorType: pbsmetrics.StoredDataErrorNetwork, + metricName: "stored_category_errors", + }, + { + description: "Update stored_request_errors counter with network label", + dataType: pbsmetrics.RequestDataType, + errorType: pbsmetrics.StoredDataErrorNetwork, + metricName: "stored_request_errors", + }, + { + description: "Update stored_video_errors counter with network label", + dataType: pbsmetrics.VideoDataType, + errorType: pbsmetrics.StoredDataErrorNetwork, + metricName: "stored_video_errors", + }, + { + description: "Update stored_account_errors counter with undefined label", + dataType: pbsmetrics.AccountDataType, + errorType: pbsmetrics.StoredDataErrorUndefined, + metricName: "stored_account_errors", + }, + { + description: "Update stored_amp_errors counter with undefined label", + dataType: pbsmetrics.AMPDataType, + errorType: pbsmetrics.StoredDataErrorUndefined, + metricName: "stored_amp_errors", + }, + { + description: "Update stored_category_errors counter with undefined label", + dataType: pbsmetrics.CategoryDataType, + errorType: pbsmetrics.StoredDataErrorUndefined, + metricName: "stored_category_errors", + }, + { + description: "Update stored_request_errors counter with undefined label", + dataType: pbsmetrics.RequestDataType, + errorType: pbsmetrics.StoredDataErrorUndefined, + metricName: "stored_request_errors", + }, + { + description: "Update stored_video_errors counter with undefined label", + dataType: pbsmetrics.VideoDataType, + errorType: pbsmetrics.StoredDataErrorUndefined, + metricName: "stored_video_errors", + }, + } + + for _, tt := range tests { + m := createMetricsForTesting() + m.RecordStoredDataError(pbsmetrics.StoredDataLabels{ + DataType: tt.dataType, + Error: tt.errorType, + }) + + var metricsCounter *prometheus.CounterVec + switch tt.dataType { + case pbsmetrics.AccountDataType: + metricsCounter = m.storedAccountErrors + case pbsmetrics.AMPDataType: + metricsCounter = m.storedAMPErrors + case pbsmetrics.CategoryDataType: + metricsCounter = m.storedCategoryErrors + case pbsmetrics.RequestDataType: + metricsCounter = m.storedRequestErrors + case pbsmetrics.VideoDataType: + metricsCounter = m.storedVideoErrors + } + + assertCounterVecValue(t, tt.description, tt.metricName, metricsCounter, + 1, + prometheus.Labels{ + storedDataErrorLabel: string(tt.errorType), + }) + } +} + func TestAdapterBidReceivedMetric(t *testing.T) { adapterName := "anyName" performTest := func(m *Metrics, hasAdm bool) { @@ -826,8 +1016,8 @@ func TestStoredReqCacheResultMetric(t *testing.T) { func TestStoredImpCacheResultMetric(t *testing.T) { m := createMetricsForTesting() - hitCount := 42 - missCount := 108 + hitCount := 41 + missCount := 107 m.RecordStoredImpCacheResult(pbsmetrics.CacheHit, hitCount) m.RecordStoredImpCacheResult(pbsmetrics.CacheMiss, missCount) @@ -843,6 +1033,26 @@ func TestStoredImpCacheResultMetric(t *testing.T) { }) } +func TestAccountCacheResultMetric(t *testing.T) { + m := createMetricsForTesting() + + hitCount := 37 + missCount := 92 + m.RecordAccountCacheResult(pbsmetrics.CacheHit, hitCount) + m.RecordAccountCacheResult(pbsmetrics.CacheMiss, missCount) + + assertCounterVecValue(t, "", "accountCacheResult:hit", m.accountCacheResult, + float64(hitCount), + prometheus.Labels{ + cacheResultLabel: string(pbsmetrics.CacheHit), + }) + assertCounterVecValue(t, "", "accountCacheResult:miss", m.accountCacheResult, + float64(missCount), + prometheus.Labels{ + cacheResultLabel: string(pbsmetrics.CacheMiss), + }) +} + func TestCookieMetric(t *testing.T) { m := createMetricsForTesting() @@ -931,13 +1141,13 @@ func TestTimeoutNotifications(t *testing.T) { m.RecordTimeoutNotice(true) m.RecordTimeoutNotice(false) - assertCounterVecValue(t, "", "timeout_notifications:ok", m.timeout_notifications, + assertCounterVecValue(t, "", "timeout_notifications:ok", m.timeoutNotifications, float64(2), prometheus.Labels{ successLabel: requestSuccessful, }) - assertCounterVecValue(t, "", "timeout_notifications:fail", m.timeout_notifications, + assertCounterVecValue(t, "", "timeout_notifications:fail", m.timeoutNotifications, float64(1), prometheus.Labels{ successLabel: requestFailed, @@ -945,6 +1155,268 @@ func TestTimeoutNotifications(t *testing.T) { } +func TestRecordDNSTime(t *testing.T) { + type testIn struct { + dnsLookupDuration time.Duration + } + type testOut struct { + expDuration float64 + expCount uint64 + } + testCases := []struct { + description string + in testIn + out testOut + }{ + { + description: "Five second DNS lookup time", + in: testIn{ + dnsLookupDuration: time.Second * 5, + }, + out: testOut{ + expDuration: 5, + expCount: 1, + }, + }, + { + description: "Zero DNS lookup time", + in: testIn{}, + out: testOut{ + expDuration: 0, + expCount: 1, + }, + }, + } + for i, test := range testCases { + pm := createMetricsForTesting() + pm.RecordDNSTime(test.in.dnsLookupDuration) + + m := dto.Metric{} + pm.dnsLookupTimer.Write(&m) + histogram := *m.GetHistogram() + + assert.Equal(t, test.out.expCount, histogram.GetSampleCount(), "[%d] Incorrect number of histogram entries. Desc: %s\n", i, test.description) + assert.Equal(t, test.out.expDuration, histogram.GetSampleSum(), "[%d] Incorrect number of histogram cumulative values. Desc: %s\n", i, test.description) + } +} + +func TestRecordAdapterConnections(t *testing.T) { + + type testIn struct { + adapterName openrtb_ext.BidderName + connWasReused bool + connWait time.Duration + } + + type testOut struct { + expectedConnReusedCount int64 + expectedConnCreatedCount int64 + expectedConnWaitCount uint64 + expectedConnWaitTime float64 + } + + testCases := []struct { + description string + in testIn + out testOut + }{ + { + description: "[1] Successful, new connection created, was idle, has connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: false, + connWait: time.Second * 5, + }, + out: testOut{ + expectedConnReusedCount: 0, + expectedConnCreatedCount: 1, + expectedConnWaitCount: 1, + expectedConnWaitTime: 5, + }, + }, + { + description: "[2] Successful, new connection created, not idle, has connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: false, + connWait: time.Second * 4, + }, + out: testOut{ + expectedConnReusedCount: 0, + expectedConnCreatedCount: 1, + expectedConnWaitCount: 1, + expectedConnWaitTime: 4, + }, + }, + { + description: "[3] Successful, was reused, was idle, no connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: true, + }, + out: testOut{ + expectedConnReusedCount: 1, + expectedConnCreatedCount: 0, + expectedConnWaitCount: 1, + expectedConnWaitTime: 0, + }, + }, + { + description: "[4] Successful, was reused, not idle, has connection wait", + in: testIn{ + adapterName: openrtb_ext.BidderAppnexus, + connWasReused: true, + connWait: time.Second * 5, + }, + out: testOut{ + expectedConnReusedCount: 1, + expectedConnCreatedCount: 0, + expectedConnWaitCount: 1, + expectedConnWaitTime: 5, + }, + }, + } + + for i, test := range testCases { + m := createMetricsForTesting() + assertDesciptions := []string{ + fmt.Sprintf("[%d] Metric: adapterReusedConnections; Desc: %s", i+1, test.description), + fmt.Sprintf("[%d] Metric: adapterCreatedConnections; Desc: %s", i+1, test.description), + fmt.Sprintf("[%d] Metric: adapterWaitConnectionCount; Desc: %s", i+1, test.description), + fmt.Sprintf("[%d] Metric: adapterWaitConnectionTime; Desc: %s", i+1, test.description), + } + + m.RecordAdapterConnections(test.in.adapterName, test.in.connWasReused, test.in.connWait) + + // Assert number of reused connections + assertCounterVecValue(t, + assertDesciptions[0], + "adapter_connection_reused", + m.adapterReusedConnections, + float64(test.out.expectedConnReusedCount), + prometheus.Labels{adapterLabel: string(test.in.adapterName)}) + + // Assert number of new created connections + assertCounterVecValue(t, + assertDesciptions[1], + "adapter_connection_created", + m.adapterCreatedConnections, + float64(test.out.expectedConnCreatedCount), + prometheus.Labels{adapterLabel: string(test.in.adapterName)}) + + // Assert connection wait time + histogram := getHistogramFromHistogramVec(m.adapterConnectionWaitTime, adapterLabel, string(test.in.adapterName)) + assert.Equal(t, test.out.expectedConnWaitCount, histogram.GetSampleCount(), assertDesciptions[2]) + assert.Equal(t, test.out.expectedConnWaitTime, histogram.GetSampleSum(), assertDesciptions[3]) + } +} + +func TestDisableAdapterConnections(t *testing.T) { + prometheusMetrics := NewMetrics(config.PrometheusMetrics{ + Port: 8080, + Namespace: "prebid", + Subsystem: "server", + }, config.DisabledMetrics{AdapterConnectionMetrics: true}) + + // Assert counter vector was not initialized + assert.Nil(t, prometheusMetrics.adapterReusedConnections, "Counter Vector adapterReusedConnections should be nil") + assert.Nil(t, prometheusMetrics.adapterCreatedConnections, "Counter Vector adapterCreatedConnections should be nil") + assert.Nil(t, prometheusMetrics.adapterConnectionWaitTime, "Counter Vector adapterConnectionWaitTime should be nil") +} + +func TestRecordRequestPrivacy(t *testing.T) { + m := createMetricsForTesting() + + // CCPA + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + CCPAEnforced: true, + CCPAProvided: true, + }) + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + CCPAEnforced: true, + CCPAProvided: false, + }) + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + CCPAEnforced: false, + CCPAProvided: true, + }) + + // COPPA + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + COPPAEnforced: true, + }) + + // LMT + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + LMTEnforced: true, + }) + + // GDPR + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionErr, + }) + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV1, + }) + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV2, + }) + m.RecordRequestPrivacy(pbsmetrics.PrivacyLabels{ + GDPREnforced: true, + GDPRTCFVersion: pbsmetrics.TCFVersionV1, + }) + + assertCounterVecValue(t, "", "privacy_ccpa", m.privacyCCPA, + float64(1), + prometheus.Labels{ + sourceLabel: sourceRequest, + optOutLabel: "true", + }) + + assertCounterVecValue(t, "", "privacy_ccpa", m.privacyCCPA, + float64(1), + prometheus.Labels{ + sourceLabel: sourceRequest, + optOutLabel: "false", + }) + + assertCounterVecValue(t, "", "privacy_coppa", m.privacyCOPPA, + float64(1), + prometheus.Labels{ + sourceLabel: sourceRequest, + }) + + assertCounterVecValue(t, "", "privacy_lmt", m.privacyLMT, + float64(1), + prometheus.Labels{ + sourceLabel: sourceRequest, + }) + + assertCounterVecValue(t, "", "privacy_tcf:err", m.privacyTCF, + float64(1), + prometheus.Labels{ + sourceLabel: sourceRequest, + versionLabel: "err", + }) + + assertCounterVecValue(t, "", "privacy_tcf:v1", m.privacyTCF, + float64(2), + prometheus.Labels{ + sourceLabel: sourceRequest, + versionLabel: "v1", + }) + + assertCounterVecValue(t, "", "privacy_tcf:v2", m.privacyTCF, + float64(1), + prometheus.Labels{ + sourceLabel: sourceRequest, + versionLabel: "v2", + }) +} + // TestRecordRequestDuplicateBidID checks RecordRequestDuplicateBidID func TestRecordRequestDuplicateBidID(t *testing.T) { m := createMetricsForTesting() @@ -1130,6 +1602,8 @@ func getHistogramFromHistogramVecByTwoKeys(histogram *prometheus.HistogramVec, l valInd := ind if ind == 1 { valInd = 0 + } else { + valInd = 1 } if m.Label[valInd].GetName() == label2Key && m.Label[valInd].GetValue() == label2Value { result = *m.GetHistogram() diff --git a/pbsmetrics/prometheus/type_conversion.go b/pbsmetrics/prometheus/type_conversion.go index ad81e84e041..5dc6e9cf29e 100644 --- a/pbsmetrics/prometheus/type_conversion.go +++ b/pbsmetrics/prometheus/type_conversion.go @@ -76,3 +76,39 @@ func requestTypesAsString() []string { } return valuesAsString } + +func storedDataTypesAsString() []string { + values := pbsmetrics.StoredDataTypes() + valuesAsString := make([]string, len(values)) + for i, v := range values { + valuesAsString[i] = string(v) + } + return valuesAsString +} + +func storedDataFetchTypesAsString() []string { + values := pbsmetrics.StoredDataFetchTypes() + valuesAsString := make([]string, len(values)) + for i, v := range values { + valuesAsString[i] = string(v) + } + return valuesAsString +} + +func storedDataErrorsAsString() []string { + values := pbsmetrics.StoredDataErrors() + valuesAsString := make([]string, len(values)) + for i, v := range values { + valuesAsString[i] = string(v) + } + return valuesAsString +} + +func tcfVersionsAsString() []string { + values := pbsmetrics.TCFVersions() + valuesAsString := make([]string, len(values)) + for i, v := range values { + valuesAsString[i] = string(v) + } + return valuesAsString +} diff --git a/prebid/prebid.go b/prebid/prebid.go deleted file mode 100644 index 68c4a48c2c8..00000000000 --- a/prebid/prebid.go +++ /dev/null @@ -1,82 +0,0 @@ -package prebid - -import ( - "net" - "net/http" - "strings" -) - -var xForwardedFor = http.CanonicalHeaderKey("X-Forwarded-For") -var xRealIP = http.CanonicalHeaderKey("X-Real-IP") -var xForwardedProto = http.CanonicalHeaderKey("X-Forwarded-Proto") - -// IsSecure attempts to detect whether the request is https -func IsSecure(r *http.Request) bool { - // lowercase for case-insensitive match for X-Forwarded-Proto header - if strings.ToLower(r.Header.Get(xForwardedProto)) == "https" { - return true - } - // ensure that URL.Scheme is lowercase (it should be "https") - if strings.ToLower(r.URL.Scheme) == "https" { - return true - } - // use strings.HasPrefix because a valid example is "HTTP/1.0" - if strings.HasPrefix(r.Proto, "HTTPS") { - return true - } - // check if TLS is not-nil as a final fallback - if r.TLS != nil { - return true - } - return false -} - -// GetIP will attempt to get the IP Address by first checking headers -// and then falling back on the RemoteAddr -func GetIP(r *http.Request) string { - // first check headers - if ip := GetForwardedIP(r); ip != "" { - return ip - } - // next try to parse the RemoteAddr. - // if err is not nil then weird hosts might appear as the ip: https://github.com/golang/go/issues/14827 - if ip, _, err := net.SplitHostPort(r.RemoteAddr); err == nil { - return ip - } - return "" -} - -// GetForwardedIP will return back X-Forwarded-For or X-Real-IP (if set) -func GetForwardedIP(r *http.Request) string { - // first attempt to parse X-Forwarded-For - if ip := getForwardedFor(r); ip != "" { - return ip - } - // if we don't have X-Forwarded-For then try X-Real-IP - if ip := getRealIP(r); ip != "" { - return ip - } - return "" -} - -// getForwardedFor will attempt to parse the X-Forwarded-For header -func getForwardedFor(r *http.Request) string { - if xff := r.Header.Get(xForwardedFor); xff != "" { - // X-Forwarded-For: client1, proxy1, proxy2 - i := strings.Index(xff, ", ") - if i == -1 { - i = len(xff) - } - return xff[:i] - } - return "" -} - -// getRealIP will attempt to parse the X-Real-IP header -// Header.Get is case-insensitive -func getRealIP(r *http.Request) string { - if xrip := r.Header.Get(xRealIP); xrip != "" { - return xrip - } - return "" -} diff --git a/prebid_cache_client/client.go b/prebid_cache_client/client.go index 314cc3e3d42..df015645145 100644 --- a/prebid_cache_client/client.go +++ b/prebid_cache_client/client.go @@ -29,8 +29,8 @@ type Client interface { // logging any relevant errors to the app logs PutJson(ctx context.Context, values []Cacheable) ([]string, []error) - // Serves the purpose of a getter that returns the host and the cache of the prebid-server URL - GetExtCacheData() (string, string) + // GetExtCacheData gets the scheme, host, and path of the externally accessible cache url. + GetExtCacheData() (scheme string, host string, path string) } type PayloadType string @@ -41,31 +41,37 @@ const ( ) type Cacheable struct { - Type PayloadType - Data json.RawMessage - TTLSeconds int64 - Key string + Type PayloadType `json:"type,omitempty"` + Data json.RawMessage `json:"value,omitempty"` + TTLSeconds int64 `json:"ttlseconds,omitempty"` + Key string `json:"key,omitempty"` + + BidID string `json:"bidid,omitempty"` // this is "/vtrack" specific + Bidder string `json:"bidder,omitempty"` // this is "/vtrack" specific + Timestamp int64 `json:"timestamp,omitempty"` // this is "/vtrack" specific } func NewClient(httpClient *http.Client, conf *config.Cache, extCache *config.ExternalCache, metrics pbsmetrics.MetricsEngine) Client { return &clientImpl{ - httpClient: httpClient, - putUrl: conf.GetBaseURL() + "/cache", - externalCacheHost: extCache.Host, - externalCachePath: extCache.Path, - metrics: metrics, + httpClient: httpClient, + putUrl: conf.GetBaseURL() + "/cache", + externalCacheScheme: extCache.Scheme, + externalCacheHost: extCache.Host, + externalCachePath: extCache.Path, + metrics: metrics, } } type clientImpl struct { - httpClient *http.Client - putUrl string - externalCacheHost string - externalCachePath string - metrics pbsmetrics.MetricsEngine + httpClient *http.Client + putUrl string + externalCacheScheme string + externalCacheHost string + externalCachePath string + metrics pbsmetrics.MetricsEngine } -func (c *clientImpl) GetExtCacheData() (string, string) { +func (c *clientImpl) GetExtCacheData() (string, string, string) { path := c.externalCachePath if path == "/" { // Only the slash for the path, remove it to empty @@ -75,7 +81,7 @@ func (c *clientImpl) GetExtCacheData() (string, string) { path = "/" + path } - return c.externalCacheHost, path + return c.externalCacheScheme, c.externalCacheHost, path } func (c *clientImpl) PutJson(ctx context.Context, values []Cacheable) (uuids []string, errs []error) { @@ -179,6 +185,25 @@ func encodeValueToBuffer(value Cacheable, leadingComma bool, buffer *bytes.Buffe buffer.WriteString(string(value.Key)) buffer.WriteString(`"`) } + + //vtrack specific + if len(value.BidID) > 0 { + buffer.WriteString(`,"bidid":"`) + buffer.WriteString(string(value.BidID)) + buffer.WriteString(`"`) + } + + if len(value.Bidder) > 0 { + buffer.WriteString(`,"bidder":"`) + buffer.WriteString(string(value.Bidder)) + buffer.WriteString(`"`) + } + + if value.Timestamp > 0 { + buffer.WriteString(`,"timestamp":`) + buffer.WriteString(strconv.FormatInt(value.Timestamp, 10)) + } + buffer.WriteByte('}') return nil } diff --git a/prebid_cache_client/client_test.go b/prebid_cache_client/client_test.go index 393aacc2dfe..9dd30d228cf 100644 --- a/prebid_cache_client/client_test.go +++ b/prebid_cache_client/client_test.go @@ -174,8 +174,11 @@ func TestEncodeValueToBuffer(t *testing.T) { Type: TypeJSON, Data: json.RawMessage(`{}`), TTLSeconds: 300, + BidID: "bid", + Bidder: "bdr", + Timestamp: 123456789, } - expected := string(`{"type":"json","ttlseconds":300,"value":{}}`) + expected := string(`{"type":"json","ttlseconds":300,"value":{},"bidid":"bid","bidder":"bdr","timestamp":123456789}`) _ = encodeValueToBuffer(testCache, false, buf) actual := buf.String() assertStringEqual(t, expected, actual) @@ -186,58 +189,80 @@ func TestEncodeValueToBuffer(t *testing.T) { func TestStripCacheHostAndPath(t *testing.T) { inCacheURL := config.Cache{ExpectedTimeMillis: 10} type aTest struct { - inExtCacheURL config.ExternalCache - expectedHost string - expectedPath string + inExtCacheURL config.ExternalCache + expectedScheme string + expectedHost string + expectedPath string } testInput := []aTest{ { inExtCacheURL: config.ExternalCache{ - Host: "prebid-server.prebid.org", - Path: "/pbcache/endpoint", + Scheme: "", + Host: "prebid-server.prebid.org", + Path: "/pbcache/endpoint", }, - expectedHost: "prebid-server.prebid.org", - expectedPath: "/pbcache/endpoint", + expectedScheme: "", + expectedHost: "prebid-server.prebid.org", + expectedPath: "/pbcache/endpoint", }, { inExtCacheURL: config.ExternalCache{ - Host: "prebidcache.net", - Path: "", + Scheme: "https", + Host: "prebid-server.prebid.org", + Path: "/pbcache/endpoint", }, - expectedHost: "prebidcache.net", - expectedPath: "", + expectedScheme: "https", + expectedHost: "prebid-server.prebid.org", + expectedPath: "/pbcache/endpoint", }, { inExtCacheURL: config.ExternalCache{ - Host: "", - Path: "", + Scheme: "", + Host: "prebidcache.net", + Path: "", }, - expectedHost: "", - expectedPath: "", + expectedScheme: "", + expectedHost: "prebidcache.net", + expectedPath: "", }, { inExtCacheURL: config.ExternalCache{ - Host: "prebid-server.prebid.org", - Path: "pbcache/endpoint", + Scheme: "", + Host: "", + Path: "", }, - expectedHost: "prebid-server.prebid.org", - expectedPath: "/pbcache/endpoint", + expectedScheme: "", + expectedHost: "", + expectedPath: "", }, { inExtCacheURL: config.ExternalCache{ - Host: "prebidcache.net", - Path: "/", + Scheme: "", + Host: "prebid-server.prebid.org", + Path: "pbcache/endpoint", }, - expectedHost: "prebidcache.net", - expectedPath: "", + expectedScheme: "", + expectedHost: "prebid-server.prebid.org", + expectedPath: "/pbcache/endpoint", + }, + { + inExtCacheURL: config.ExternalCache{ + Scheme: "", + Host: "prebidcache.net", + Path: "/", + }, + expectedScheme: "", + expectedHost: "prebidcache.net", + expectedPath: "", }, } for _, test := range testInput { cacheClient := NewClient(&http.Client{}, &inCacheURL, &test.inExtCacheURL, &metricsConf.DummyMetricsEngine{}) - cHost, cPath := cacheClient.GetExtCacheData() + scheme, host, path := cacheClient.GetExtCacheData() - assert.Equal(t, test.expectedHost, cHost) - assert.Equal(t, test.expectedPath, cPath) + assert.Equal(t, test.expectedScheme, scheme) + assert.Equal(t, test.expectedHost, host) + assert.Equal(t, test.expectedPath, path) } } diff --git a/privacy/ccpa/consentwriter.go b/privacy/ccpa/consentwriter.go new file mode 100644 index 00000000000..4ef412fd3ef --- /dev/null +++ b/privacy/ccpa/consentwriter.go @@ -0,0 +1,25 @@ +package ccpa + +import ( + "github.com/PubMatic-OpenWrap/openrtb" +) + +// ConsentWriter implements the PolicyWriter interface for CCPA. +type ConsentWriter struct { + Consent string +} + +// Write mutates an OpenRTB bid request with the CCPA consent string. +func (c ConsentWriter) Write(req *openrtb.BidRequest) error { + if req == nil { + return nil + } + + regs, err := buildRegs(c.Consent, req.Regs) + if err != nil { + return err + } + req.Regs = regs + + return nil +} diff --git a/privacy/ccpa/consentwriter_test.go b/privacy/ccpa/consentwriter_test.go new file mode 100644 index 00000000000..1e491d9d167 --- /dev/null +++ b/privacy/ccpa/consentwriter_test.go @@ -0,0 +1,51 @@ +package ccpa + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/stretchr/testify/assert" +) + +func TestConsentWriter(t *testing.T) { + consent := "anyConsent" + testCases := []struct { + description string + request *openrtb.BidRequest + expected *openrtb.BidRequest + expectedError bool + }{ + { + description: "Nil Request", + request: nil, + expected: nil, + }, + { + description: "Success", + request: &openrtb.BidRequest{}, + expected: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, + }, + }, + { + description: "Error With Regs.Ext - Does Not Mutate", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`malformed}`)}, + }, + expectedError: true, + expected: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`malformed}`)}, + }, + }, + } + + for _, test := range testCases { + writer := ConsentWriter{consent} + + err := writer.Write(test.request) + + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, test.request, test.description) + } +} diff --git a/privacy/ccpa/parsedpolicy.go b/privacy/ccpa/parsedpolicy.go new file mode 100644 index 00000000000..52977104716 --- /dev/null +++ b/privacy/ccpa/parsedpolicy.go @@ -0,0 +1,137 @@ +package ccpa + +import ( + "errors" + "fmt" + + "github.com/PubMatic-OpenWrap/prebid-server/errortypes" +) + +const ( + ccpaVersion1 = '1' + ccpaYes = 'Y' + ccpaNo = 'N' + ccpaNotApplicable = '-' +) + +const ( + indexVersion = 0 + indexExplicitNotice = 1 + indexOptOutSale = 2 + indexLSPACoveredTransaction = 3 +) + +const allBiddersMarker = "*" + +// ValidateConsent returns true if the consent string is empty or valid per the IAB CCPA spec. +func ValidateConsent(consent string) bool { + _, err := parseConsent(consent) + return err == nil +} + +// ParsedPolicy represents parsed and validated CCPA regulatory information. Use this struct +// to make enforcement decisions. +type ParsedPolicy struct { + consentSpecified bool + consentOptOutSale bool + noSaleForAllBidders bool + noSaleSpecificBidders map[string]struct{} +} + +// Parse returns a parsed and validated ParsedPolicy intended for use in enforcement decisions. +func (p Policy) Parse(validBidders map[string]struct{}) (ParsedPolicy, error) { + consentOptOut, err := parseConsent(p.Consent) + if err != nil { + msg := fmt.Sprintf("request.regs.ext.us_privacy %s", err.Error()) + return ParsedPolicy{}, &errortypes.InvalidPrivacyConsent{Message: msg} + } + + noSaleForAllBidders, noSaleSpecificBidders, err := parseNoSaleBidders(p.NoSaleBidders, validBidders) + if err != nil { + return ParsedPolicy{}, fmt.Errorf("request.ext.prebid.nosale is invalid: %s", err.Error()) + } + + return ParsedPolicy{ + consentSpecified: p.Consent != "", + consentOptOutSale: consentOptOut, + noSaleForAllBidders: noSaleForAllBidders, + noSaleSpecificBidders: noSaleSpecificBidders, + }, nil +} + +func parseConsent(consent string) (consentOptOutSale bool, err error) { + if consent == "" { + return false, nil + } + + if len(consent) != 4 { + return false, errors.New("must contain 4 characters") + } + + if consent[indexVersion] != ccpaVersion1 { + return false, errors.New("must specify version 1") + } + + var c byte + + c = consent[indexExplicitNotice] + if c != ccpaNo && c != ccpaYes && c != ccpaNotApplicable { + return false, errors.New("must specify 'N', 'Y', or '-' for the explicit notice") + } + + c = consent[indexOptOutSale] + if c != ccpaNo && c != ccpaYes && c != ccpaNotApplicable { + return false, errors.New("must specify 'N', 'Y', or '-' for the opt-out sale") + } + + c = consent[indexLSPACoveredTransaction] + if c != ccpaNo && c != ccpaYes && c != ccpaNotApplicable { + return false, errors.New("must specify 'N', 'Y', or '-' for the limited service provider agreement") + } + + return consent[indexOptOutSale] == ccpaYes, nil +} + +func parseNoSaleBidders(noSaleBidders []string, validBidders map[string]struct{}) (noSaleForAllBidders bool, noSaleSpecificBidders map[string]struct{}, err error) { + noSaleSpecificBidders = make(map[string]struct{}) + + if len(noSaleBidders) == 1 && noSaleBidders[0] == allBiddersMarker { + noSaleForAllBidders = true + return + } + + for _, bidder := range noSaleBidders { + if bidder == allBiddersMarker { + err = errors.New("can only specify all bidders if no other bidders are provided") + return + } + + if _, exists := validBidders[bidder]; exists { + noSaleSpecificBidders[bidder] = struct{}{} + } else { + err = fmt.Errorf("unrecognized bidder '%s'", bidder) + return + } + } + + return +} + +// CanEnforce returns true when consent is specifically provided by the publisher, as opposed to an empty string. +func (p ParsedPolicy) CanEnforce() bool { + return p.consentSpecified +} + +func (p ParsedPolicy) isNoSaleForBidder(bidder string) bool { + if p.noSaleForAllBidders { + return true + } + + _, exists := p.noSaleSpecificBidders[bidder] + return exists +} + +// ShouldEnforce returns true when the opt-out signal is explicitly detected. +func (p ParsedPolicy) ShouldEnforce(bidder string) bool { + return !p.isNoSaleForBidder(bidder) && p.consentOptOutSale +} diff --git a/privacy/ccpa/parsedpolicy_test.go b/privacy/ccpa/parsedpolicy_test.go new file mode 100644 index 00000000000..4fa9f92684d --- /dev/null +++ b/privacy/ccpa/parsedpolicy_test.go @@ -0,0 +1,391 @@ +package ccpa + +import ( + "testing" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" +) + +func TestValidateConsent(t *testing.T) { + testCases := []struct { + description string + consent string + expected bool + }{ + { + description: "Empty String", + consent: "", + expected: true, + }, + { + description: "Valid Consent With Opt Out", + consent: "1NYN", + expected: true, + }, + { + description: "Valid Consent Without Opt Out", + consent: "1NNN", + expected: true, + }, + { + description: "Invalid", + consent: "malformed", + expected: false, + }, + } + + for _, test := range testCases { + result := ValidateConsent(test.consent) + assert.Equal(t, test.expected, result, test.description) + } +} + +func TestParse(t *testing.T) { + validBidders := map[string]struct{}{"a": {}} + + testCases := []struct { + description string + consent string + noSaleBidders []string + expectedPolicy ParsedPolicy + expectedError string + }{ + { + description: "Consent Error", + consent: "malformed", + noSaleBidders: []string{}, + expectedPolicy: ParsedPolicy{}, + expectedError: "request.regs.ext.us_privacy must contain 4 characters", + }, + { + description: "No Sale Error", + consent: "1NYN", + noSaleBidders: []string{"b"}, + expectedPolicy: ParsedPolicy{}, + expectedError: "request.ext.prebid.nosale is invalid: unrecognized bidder 'b'", + }, + { + description: "Success", + consent: "1NYN", + noSaleBidders: []string{"a"}, + expectedPolicy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: true, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{"a": {}}, + }, + }, + } + + for _, test := range testCases { + policy := Policy{test.consent, test.noSaleBidders} + + result, err := policy.Parse(validBidders) + + if test.expectedError == "" { + assert.NoError(t, err, test.description) + } else { + assert.EqualError(t, err, test.expectedError, test.description) + } + + assert.Equal(t, test.expectedPolicy, result, test.description) + } +} + +func TestParseConsent(t *testing.T) { + testCases := []struct { + description string + consent string + expectedResult bool + expectedError string + }{ + { + description: "Valid", + consent: "1NYN", + expectedResult: true, + }, + { + description: "Valid - Not Sale", + consent: "1NNN", + expectedResult: false, + }, + { + description: "Valid - Not Applicable", + consent: "1---", + expectedResult: false, + }, + { + description: "Valid - Empty", + consent: "", + expectedResult: false, + }, + { + description: "Wrong Length", + consent: "1NY", + expectedResult: false, + expectedError: "must contain 4 characters", + }, + { + description: "Wrong Version", + consent: "2---", + expectedResult: false, + expectedError: "must specify version 1", + }, + { + description: "Explicit Notice Char", + consent: "1X--", + expectedResult: false, + expectedError: "must specify 'N', 'Y', or '-' for the explicit notice", + }, + { + description: "Invalid Explicit Notice Case", + consent: "1y--", + expectedResult: false, + expectedError: "must specify 'N', 'Y', or '-' for the explicit notice", + }, + { + description: "Invalid Opt-Out Sale Char", + consent: "1-X-", + expectedResult: false, + expectedError: "must specify 'N', 'Y', or '-' for the opt-out sale", + }, + { + description: "Invalid Opt-Out Sale Case", + consent: "1-y-", + expectedResult: false, + expectedError: "must specify 'N', 'Y', or '-' for the opt-out sale", + }, + { + description: "Invalid LSPA Char", + consent: "1--X", + expectedResult: false, + expectedError: "must specify 'N', 'Y', or '-' for the limited service provider agreement", + }, + { + description: "Invalid LSPA Case", + consent: "1--y", + expectedResult: false, + expectedError: "must specify 'N', 'Y', or '-' for the limited service provider agreement", + }, + } + + for _, test := range testCases { + result, err := parseConsent(test.consent) + + if test.expectedError == "" { + assert.NoError(t, err, test.description) + } else { + assert.EqualError(t, err, test.expectedError, test.description) + } + + assert.Equal(t, test.expectedResult, result, test.description) + } +} + +func TestParseNoSaleBidders(t *testing.T) { + testCases := []struct { + description string + noSaleBidders []string + validBidders []string + expectedNoSaleForAllBidders bool + expectedNoSaleSpecificBidders map[string]struct{} + expectedError string + }{ + { + description: "Valid - No Bidders", + noSaleBidders: []string{}, + validBidders: []string{"a"}, + expectedNoSaleForAllBidders: false, + expectedNoSaleSpecificBidders: map[string]struct{}{}, + }, + { + description: "Valid - 1 Bidder", + noSaleBidders: []string{"a"}, + validBidders: []string{"a"}, + expectedNoSaleForAllBidders: false, + expectedNoSaleSpecificBidders: map[string]struct{}{"a": {}}, + }, + { + description: "Valid - 1+ Bidders", + noSaleBidders: []string{"a", "b"}, + validBidders: []string{"a", "b"}, + expectedNoSaleForAllBidders: false, + expectedNoSaleSpecificBidders: map[string]struct{}{"a": {}, "b": {}}, + }, + { + description: "Valid - All Bidders", + noSaleBidders: []string{"*"}, + validBidders: []string{"a"}, + expectedNoSaleForAllBidders: true, + expectedNoSaleSpecificBidders: map[string]struct{}{}, + }, + { + description: "Bidder Not Valid", + noSaleBidders: []string{"b"}, + validBidders: []string{"a"}, + expectedError: "unrecognized bidder 'b'", + expectedNoSaleForAllBidders: false, + expectedNoSaleSpecificBidders: map[string]struct{}{}, + }, + { + description: "All Bidder Mixed With Other Bidders Is Invalid", + noSaleBidders: []string{"*", "a"}, + validBidders: []string{"a"}, + expectedError: "can only specify all bidders if no other bidders are provided", + expectedNoSaleForAllBidders: false, + expectedNoSaleSpecificBidders: map[string]struct{}{}, + }, + { + description: "Valid Bidders Case Sensitive", + noSaleBidders: []string{"a"}, + validBidders: []string{"A"}, + expectedError: "unrecognized bidder 'a'", + expectedNoSaleForAllBidders: false, + expectedNoSaleSpecificBidders: map[string]struct{}{}, + }, + } + + for _, test := range testCases { + validBiddersMap := make(map[string]struct{}) + for _, v := range test.validBidders { + validBiddersMap[v] = struct{}{} + } + + resultNoSaleForAllBidders, resultNoSaleSpecificBidders, err := parseNoSaleBidders(test.noSaleBidders, validBiddersMap) + + if test.expectedError == "" { + assert.NoError(t, err, test.description+":err") + } else { + assert.EqualError(t, err, test.expectedError, test.description+":err") + } + + assert.Equal(t, test.expectedNoSaleForAllBidders, resultNoSaleForAllBidders, test.description+":allBidders") + assert.Equal(t, test.expectedNoSaleSpecificBidders, resultNoSaleSpecificBidders, test.description+":specificBidders") + } +} + +func TestCanEnforce(t *testing.T) { + testCases := []struct { + description string + policy ParsedPolicy + expected bool + }{ + { + description: "Specified", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: false, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{}, + }, + expected: true, + }, + { + description: "Not Specified", + policy: ParsedPolicy{ + consentSpecified: false, + consentOptOutSale: false, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{}, + }, + expected: false, + }, + } + + for _, test := range testCases { + result := test.policy.CanEnforce() + assert.Equal(t, test.expected, result, test.description) + } +} + +func TestShouldEnforce(t *testing.T) { + testCases := []struct { + description string + policy ParsedPolicy + bidder string + expected bool + }{ + { + description: "Not Enforced - All Bidders No Sale", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: true, + noSaleForAllBidders: true, + noSaleSpecificBidders: map[string]struct{}{}, + }, + bidder: "a", + expected: false, + }, + { + description: "Not Enforced - Specific Bidders No Sale", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: true, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{"a": {}}, + }, + bidder: "a", + expected: false, + }, + { + description: "Not Enforced - No Bidder No Sale", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: false, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{}, + }, + bidder: "a", + expected: false, + }, + { + description: "Not Enforced - No Sale Case Sensitive", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: false, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{"A": {}}, + }, + bidder: "a", + expected: false, + }, + { + description: "Enforced - No Bidder No Sale", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: true, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{}, + }, + bidder: "a", + expected: true, + }, + { + description: "Enforced - No Sale Case Sensitive", + policy: ParsedPolicy{ + consentSpecified: true, + consentOptOutSale: true, + noSaleForAllBidders: false, + noSaleSpecificBidders: map[string]struct{}{"A": {}}, + }, + bidder: "a", + expected: true, + }, + } + + for _, test := range testCases { + result := test.policy.ShouldEnforce(test.bidder) + assert.Equal(t, test.expected, result, test.description) + } +} + +type mockPolicWriter struct { + mock.Mock +} + +func (m *mockPolicWriter) Write(req *openrtb.BidRequest) error { + args := m.Called(req) + return args.Error(0) +} diff --git a/privacy/ccpa/policy.go b/privacy/ccpa/policy.go index d4299af8cf2..3f5dd25c6bc 100644 --- a/privacy/ccpa/policy.go +++ b/privacy/ccpa/policy.go @@ -9,139 +9,190 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" ) -// Policy represents the CCPA regulation for an OpenRTB bid request. +// Policy represents the CCPA regulatory information from an OpenRTB bid request. type Policy struct { - Value string + Consent string + NoSaleBidders []string } -// ReadPolicy extracts the CCPA regulation policy from an OpenRTB request. -func ReadPolicy(req *openrtb.BidRequest) (Policy, error) { - policy := Policy{} +// ReadFromRequest extracts the CCPA regulatory information from an OpenRTB bid request. +func ReadFromRequest(req *openrtb.BidRequest) (Policy, error) { + var consent string + var noSaleBidders []string - if req != nil && req.Regs != nil && len(req.Regs.Ext) > 0 { + if req == nil { + return Policy{}, nil + } + + // Read consent from request.regs.ext + if req.Regs != nil && len(req.Regs.Ext) > 0 { var ext openrtb_ext.ExtRegs if err := json.Unmarshal(req.Regs.Ext, &ext); err != nil { - return policy, err + return Policy{}, fmt.Errorf("error reading request.regs.ext: %s", err) } - policy.Value = ext.USPrivacy + consent = ext.USPrivacy } - return policy, nil + // Read no sale bidders from request.ext.prebid + if len(req.Ext) > 0 { + var ext openrtb_ext.ExtRequest + if err := json.Unmarshal(req.Ext, &ext); err != nil { + return Policy{}, fmt.Errorf("error reading request.ext.prebid: %s", err) + } + noSaleBidders = ext.Prebid.NoSale + } + + return Policy{consent, noSaleBidders}, nil } -// Write mutates an OpenRTB bid request with the context of the CCPA policy. +// Write mutates an OpenRTB bid request with the CCPA regulatory information. func (p Policy) Write(req *openrtb.BidRequest) error { - if p.Value == "" { - return clearPolicy(req) - } - if req == nil { return nil } - if req.Regs == nil { - req.Regs = &openrtb.Regs{} + regs, err := buildRegs(p.Consent, req.Regs) + if err != nil { + return err } - - if req.Regs.Ext == nil { - ext, err := json.Marshal(openrtb_ext.ExtRegs{USPrivacy: p.Value}) - if err == nil { - req.Regs.Ext = ext - } + ext, err := buildExt(p.NoSaleBidders, req.Ext) + if err != nil { return err } - var extMap map[string]interface{} - err := json.Unmarshal(req.Regs.Ext, &extMap) - if err == nil { - extMap["us_privacy"] = p.Value - ext, err := json.Marshal(extMap) - if err == nil { - req.Regs.Ext = ext - } + req.Regs = regs + req.Ext = ext + return nil +} + +func buildRegs(consent string, regs *openrtb.Regs) (*openrtb.Regs, error) { + if consent == "" { + return buildRegsClear(regs) } - return err + return buildRegsWrite(consent, regs) } -func clearPolicy(req *openrtb.BidRequest) error { - if req == nil { - return nil +func buildRegsClear(regs *openrtb.Regs) (*openrtb.Regs, error) { + if regs == nil || len(regs.Ext) == 0 { + return regs, nil } - if req.Regs == nil { - return nil + var extMap map[string]interface{} + if err := json.Unmarshal(regs.Ext, &extMap); err != nil { + return nil, err } - if len(req.Regs.Ext) == 0 { - return nil + delete(extMap, "us_privacy") + + // Remove entire ext if it's now empty + if len(extMap) == 0 { + regsResult := *regs + regsResult.Ext = nil + return ®sResult, nil } - var extMap map[string]interface{} - err := json.Unmarshal(req.Regs.Ext, &extMap) + // Marshal ext if there are still other fields + var regsResult openrtb.Regs + ext, err := json.Marshal(extMap) if err == nil { - delete(extMap, "us_privacy") - if len(extMap) == 0 { - req.Regs.Ext = nil - } else { - ext, err := json.Marshal(extMap) - if err == nil { - req.Regs.Ext = ext - } - return err - } + regsResult = *regs + regsResult.Ext = ext } - - return err + return ®sResult, err } -// Validate returns an error if the CCPA policy does not adhere to the IAB spec. -func (p Policy) Validate() error { - if err := ValidateConsent(p.Value); err != nil { - return fmt.Errorf("request.regs.ext.us_privacy %s", err.Error()) +func buildRegsWrite(consent string, regs *openrtb.Regs) (*openrtb.Regs, error) { + if regs == nil { + return marshalRegsExt(openrtb.Regs{}, openrtb_ext.ExtRegs{USPrivacy: consent}) } - return nil + if regs.Ext == nil { + return marshalRegsExt(*regs, openrtb_ext.ExtRegs{USPrivacy: consent}) + } + + var extMap map[string]interface{} + if err := json.Unmarshal(regs.Ext, &extMap); err != nil { + return nil, err + } + + extMap["us_privacy"] = consent + return marshalRegsExt(*regs, extMap) } -// ValidateConsent returns an error if the CCPA consent string does not adhere to the IAB spec. -func ValidateConsent(consent string) error { - if consent == "" { - return nil +func marshalRegsExt(regs openrtb.Regs, ext interface{}) (*openrtb.Regs, error) { + extJSON, err := json.Marshal(ext) + if err == nil { + regs.Ext = extJSON } + return ®s, err +} - if len(consent) != 4 { - return errors.New("must contain 4 characters") +func buildExt(noSaleBidders []string, ext json.RawMessage) (json.RawMessage, error) { + if len(noSaleBidders) == 0 { + return buildExtClear(ext) } + return buildExtWrite(noSaleBidders, ext) +} - if consent[0] != '1' { - return errors.New("must specify version 1") +func buildExtClear(ext json.RawMessage) (json.RawMessage, error) { + if len(ext) == 0 { + return ext, nil } - var c byte + var extMap map[string]interface{} + if err := json.Unmarshal(ext, &extMap); err != nil { + return nil, err + } - c = consent[1] - if c != 'N' && c != 'Y' && c != '-' { - return errors.New("must specify 'N', 'Y', or '-' for the explicit notice") + prebidExt, exists := extMap["prebid"] + if !exists { + return ext, nil } - c = consent[2] - if c != 'N' && c != 'Y' && c != '-' { - return errors.New("must specify 'N', 'Y', or '-' for the opt-out sale") + // Verify prebid is an object + prebidExtMap, ok := prebidExt.(map[string]interface{}) + if !ok { + return nil, errors.New("request.ext.prebid is not a json object") } - c = consent[3] - if c != 'N' && c != 'Y' && c != '-' { - return errors.New("must specify 'N', 'Y', or '-' for the limited service provider agreement") + // Remove no sale member + delete(prebidExtMap, "nosale") + if len(prebidExtMap) == 0 { + delete(extMap, "prebid") } - return nil + // Remove entire ext if it's empty + if len(extMap) == 0 { + return nil, nil + } + + return json.Marshal(extMap) } -// ShouldEnforce returns true when the opt-out signal is explicitly detected. -func (p Policy) ShouldEnforce() bool { - if err := p.Validate(); err != nil { - return false +func buildExtWrite(noSaleBidders []string, ext json.RawMessage) (json.RawMessage, error) { + if len(ext) == 0 { + return json.Marshal(openrtb_ext.ExtRequest{Prebid: openrtb_ext.ExtRequestPrebid{NoSale: noSaleBidders}}) + } + + var extMap map[string]interface{} + if err := json.Unmarshal(ext, &extMap); err != nil { + return nil, err + } + + var prebidExt map[string]interface{} + if prebidExtInterface, exists := extMap["prebid"]; exists { + // Reference Existing Prebid Ext Map + if prebidExtMap, ok := prebidExtInterface.(map[string]interface{}); ok { + prebidExt = prebidExtMap + } else { + return nil, errors.New("request.ext.prebid is not a json object") + } + } else { + // Create New Empty Prebid Ext Map + prebidExt = make(map[string]interface{}) + extMap["prebid"] = prebidExt } - return p.Value != "" && p.Value[2] == 'Y' + prebidExt["nosale"] = noSaleBidders + return json.Marshal(extMap) } diff --git a/privacy/ccpa/policy_test.go b/privacy/ccpa/policy_test.go index 647f85481b3..c1fdd9cd903 100644 --- a/privacy/ccpa/policy_test.go +++ b/privacy/ccpa/policy_test.go @@ -8,7 +8,7 @@ import ( "github.com/stretchr/testify/assert" ) -func TestRead(t *testing.T) { +func TestReadFromRequest(t *testing.T) { testCases := []struct { description string request *openrtb.BidRequest @@ -18,83 +18,146 @@ func TestRead(t *testing.T) { { description: "Success", request: &openrtb.BidRequest{ - Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"ABC"}`), - }, + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), }, expectedPolicy: Policy{ - Value: "ABC", + Consent: "ABC", + NoSaleBidders: []string{"a", "b"}, }, }, { - description: "Empty - No Request", + description: "Nil Request", request: nil, expectedPolicy: Policy{ - Value: "", + Consent: "", + NoSaleBidders: nil, }, }, { - description: "Empty - No Regs", + description: "Nil Regs", request: &openrtb.BidRequest{ Regs: nil, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), }, expectedPolicy: Policy{ - Value: "", + Consent: "", + NoSaleBidders: []string{"a", "b"}, }, }, { - description: "Empty - No Ext", + description: "Nil Regs.Ext", request: &openrtb.BidRequest{ Regs: &openrtb.Regs{}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), }, expectedPolicy: Policy{ - Value: "", + Consent: "", + NoSaleBidders: []string{"a", "b"}, }, }, { - description: "Empty - No Value", + description: "Empty Regs.Ext", request: &openrtb.BidRequest{ - Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"anythingElse":"42"}`), - }, + Regs: &openrtb.Regs{Ext: json.RawMessage(`{}`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), }, expectedPolicy: Policy{ - Value: "", + Consent: "", + NoSaleBidders: []string{"a", "b"}, }, }, { - description: "Serialization Issue", + description: "Missing Regs.Ext USPrivacy Value", request: &openrtb.BidRequest{ - Regs: &openrtb.Regs{ - Ext: json.RawMessage(`malformed`), - }, + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"anythingElse":"42"}`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), + }, + expectedPolicy: Policy{ + Consent: "", + NoSaleBidders: []string{"a", "b"}, + }, + }, + { + description: "Malformed Regs.Ext", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`malformed`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), + }, + expectedError: true, + }, + { + description: "Invalid Regs.Ext Type", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":123`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), + }, + expectedError: true, + }, + { + description: "Nil Ext", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + Ext: nil, + }, + expectedPolicy: Policy{ + Consent: "ABC", + NoSaleBidders: nil, + }, + }, + { + description: "Empty Ext", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + Ext: json.RawMessage(`{}`), + }, + expectedPolicy: Policy{ + Consent: "ABC", + NoSaleBidders: nil, + }, + }, + { + description: "Missing Ext.Prebid No Sale Value", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + Ext: json.RawMessage(`{"anythingElse":"42"}`), + }, + expectedPolicy: Policy{ + Consent: "ABC", + NoSaleBidders: nil, + }, + }, + { + description: "Malformed Ext", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + Ext: json.RawMessage(`malformed`), + }, + expectedError: true, + }, + { + description: "Invalid Ext.Prebid.NoSale Type", + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":"wrongtype"}}`), }, expectedError: true, }, { description: "Injection Attack", request: &openrtb.BidRequest{ - Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), - }, + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`)}, }, expectedPolicy: Policy{ - Value: "1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"", + Consent: "1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"", }, }, } for _, test := range testCases { - - p, e := ReadPolicy(test.request) - - if test.expectedError { - assert.Error(t, e, test.description) - } else { - assert.NoError(t, e, test.description) - } - - assert.Equal(t, test.expectedPolicy, p, test.description) + result, err := ReadFromRequest(test.request) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expectedPolicy, result, test.description) } } @@ -107,313 +170,422 @@ func TestWrite(t *testing.T) { expectedError bool }{ { - description: "Disabled", - policy: Policy{Value: ""}, - request: &openrtb.BidRequest{}, - expected: &openrtb.BidRequest{}, - }, - { - description: "Disabled - Nil Request", - policy: Policy{Value: ""}, + description: "Nil Request", + policy: Policy{Consent: "anyConsent", NoSaleBidders: []string{"a", "b"}}, request: nil, expected: nil, }, { - description: "Disabled - Empty Regs.Ext", - policy: Policy{Value: ""}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{}}, + description: "Success", + policy: Policy{Consent: "anyConsent", NoSaleBidders: []string{"a", "b"}}, + request: &openrtb.BidRequest{}, + expected: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, + Ext: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), + }, }, { - description: "Disabled - Remove From Request", - policy: Policy{Value: ""}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"toBeRemoved"}`)}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{}}, + description: "Error Regs.Ext - No Partial Update To Request", + policy: Policy{Consent: "anyConsent", NoSaleBidders: []string{"a", "b"}}, + request: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`malformed}`)}, + }, + expectedError: true, + expected: &openrtb.BidRequest{ + Regs: &openrtb.Regs{Ext: json.RawMessage(`malformed}`)}, + }, }, { - description: "Disabled - Remove From Request, Leave Other req Values", - policy: Policy{Value: ""}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - COPPA: 42, - Ext: json.RawMessage(`{"us_privacy":"toBeRemoved"}`)}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - COPPA: 42}}, + description: "Error Ext - No Partial Update To Request", + policy: Policy{Consent: "anyConsent", NoSaleBidders: []string{"a", "b"}}, + request: &openrtb.BidRequest{ + Ext: json.RawMessage(`malformed}`), + }, + expectedError: true, + expected: &openrtb.BidRequest{ + Ext: json.RawMessage(`malformed}`), + }, }, + } + + for _, test := range testCases { + err := test.policy.Write(test.request) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, test.request, test.description) + } +} + +func TestBuildRegs(t *testing.T) { + testCases := []struct { + description string + consent string + regs *openrtb.Regs + expected *openrtb.Regs + expectedError bool + }{ { - description: "Disabled - Remove From Request, Leave Other req.ext Values", - policy: Policy{Value: ""}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any","us_privacy":"toBeRemoved"}`)}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any"}`)}}, + description: "Clear", + consent: "", + regs: &openrtb.Regs{ + Ext: json.RawMessage(`{"us_privacy":"ABC"}`), + }, + expected: &openrtb.Regs{}, }, { - description: "Enabled - Nil Request", - policy: Policy{Value: "anyValue"}, - request: nil, - expected: nil, + description: "Clear - Error", + consent: "", + regs: &openrtb.Regs{ + Ext: json.RawMessage(`malformed`), + }, + expectedError: true, }, { - description: "Enabled With Nil Request Regs Object", - policy: Policy{Value: "anyValue"}, - request: &openrtb.BidRequest{}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"anyValue"}`)}}, + description: "Write", + consent: "anyConsent", + regs: nil, + expected: &openrtb.Regs{ + Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`), + }, }, { - description: "Enabled With Nil Request Regs Ext Object", - policy: Policy{Value: "anyValue"}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"anyValue"}`)}}, + description: "Write - Error", + consent: "anyConsent", + regs: &openrtb.Regs{ + Ext: json.RawMessage(`malformed`), + }, + expectedError: true, }, + } + + for _, test := range testCases { + result, err := buildRegs(test.consent, test.regs) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, result, test.description) + } +} + +func TestBuildRegsClear(t *testing.T) { + testCases := []struct { + description string + regs *openrtb.Regs + expected *openrtb.Regs + expectedError bool + }{ { - description: "Enabled With Existing Request Regs Ext Object - Doesn't Overwrite", - policy: Policy{Value: "anyValue"}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any"}`)}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any","us_privacy":"anyValue"}`)}}, + description: "Nil Regs", + regs: nil, + expected: nil, }, { - description: "Enabled With Existing Request Regs Ext Object - Overwrites", - policy: Policy{Value: "anyValue"}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any","us_privacy":"toBeOverwritten"}`)}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any","us_privacy":"anyValue"}`)}}, + description: "Nil Regs.Ext", + regs: &openrtb.Regs{Ext: nil}, + expected: &openrtb.Regs{Ext: nil}, }, { - description: "Enabled With Existing Malformed Request Regs Ext Object", - policy: Policy{Value: "anyValue"}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`malformed`)}}, - expectedError: true, + description: "Empty Regs.Ext", + regs: &openrtb.Regs{Ext: json.RawMessage(`{}`)}, + expected: &openrtb.Regs{}, }, { - description: "Injection Attack With Nil Request Regs Object", - policy: Policy{Value: "1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}, - request: &openrtb.BidRequest{}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), - }}, + description: "Removes Regs.Ext Entirely", + regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + expected: &openrtb.Regs{}, + }, + { + description: "Leaves Other Regs.Ext Values", + regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC", "other":"any"}`)}, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"other":"any"}`)}, }, { - description: "Injection Attack With Nil Request Regs Ext Object", - policy: Policy{Value: "1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{}}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"us_privacy":"1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), - }}, + description: "Invalid Regs.Ext Type - Still Cleared", + regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":123}`)}, + expected: &openrtb.Regs{}, }, { - description: "Injection Attack With Existing Request Regs Ext Object", - policy: Policy{Value: "1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}, - request: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any"}`), - }}, - expected: &openrtb.BidRequest{Regs: &openrtb.Regs{ - Ext: json.RawMessage(`{"existing":"any","us_privacy":"1YYY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), - }}, + description: "Malformed Regs.Ext", + regs: &openrtb.Regs{Ext: json.RawMessage(`malformed`)}, + expectedError: true, }, } for _, test := range testCases { - err := test.policy.Write(test.request) - - if test.expectedError { - assert.Error(t, err, test.description) - } else { - assert.NoError(t, err, test.description) - assert.Equal(t, test.expected, test.request, test.description) - } + result, err := buildRegsClear(test.regs) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, result, test.description) } } -func TestValidate(t *testing.T) { +func TestBuildRegsWrite(t *testing.T) { testCases := []struct { description string - policy Policy - expectedError string + consent string + regs *openrtb.Regs + expected *openrtb.Regs + expectedError bool }{ { - description: "Valid", - policy: Policy{Value: "1NYN"}, - expectedError: "", + description: "Nil Regs", + consent: "anyConsent", + regs: nil, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, }, { - description: "Valid - Not Applicable", - policy: Policy{Value: "1---"}, - expectedError: "", + description: "Nil Regs.Ext", + consent: "anyConsent", + regs: &openrtb.Regs{Ext: nil}, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, }, { - description: "Valid - Empty", - policy: Policy{Value: ""}, - expectedError: "", + description: "Empty Regs.Ext", + consent: "anyConsent", + regs: &openrtb.Regs{Ext: json.RawMessage(`{}`)}, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, }, { - description: "Invalid Length", - policy: Policy{Value: "1NY"}, - expectedError: "request.regs.ext.us_privacy must contain 4 characters", + description: "Overwrites Existing", + consent: "anyConsent", + regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"ABC"}`)}, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, }, { - description: "Invalid Version", - policy: Policy{Value: "2---"}, - expectedError: "request.regs.ext.us_privacy must specify version 1", + description: "Leaves Other Ext Values", + consent: "anyConsent", + regs: &openrtb.Regs{Ext: json.RawMessage(`{"other":"any"}`)}, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"other":"any","us_privacy":"anyConsent"}`)}, }, { - description: "Invalid Explicit Notice Char", - policy: Policy{Value: "1X--"}, - expectedError: "request.regs.ext.us_privacy must specify 'N', 'Y', or '-' for the explicit notice", + description: "Invalid Regs.Ext Type - Still Overwrites", + consent: "anyConsent", + regs: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":123}`)}, + expected: &openrtb.Regs{Ext: json.RawMessage(`{"us_privacy":"anyConsent"}`)}, }, { - description: "Invalid Explicit Notice Case", - policy: Policy{Value: "1y--"}, - expectedError: "request.regs.ext.us_privacy must specify 'N', 'Y', or '-' for the explicit notice", + description: "Malformed Regs.Ext", + consent: "anyConsent", + regs: &openrtb.Regs{Ext: json.RawMessage(`malformed`)}, + expectedError: true, }, + } + + for _, test := range testCases { + result, err := buildRegsWrite(test.consent, test.regs) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, result, test.description) + } +} + +func TestBuildExt(t *testing.T) { + testCases := []struct { + description string + noSaleBidders []string + ext json.RawMessage + expected json.RawMessage + expectedError bool + }{ { - description: "Invalid Opt-Out Sale Char", - policy: Policy{Value: "1-X-"}, - expectedError: "request.regs.ext.us_privacy must specify 'N', 'Y', or '-' for the opt-out sale", + description: "Clear - Nil", + noSaleBidders: nil, + ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), + expected: nil, }, { - description: "Invalid Opt-Out Sale Case", - policy: Policy{Value: "1-y-"}, - expectedError: "request.regs.ext.us_privacy must specify 'N', 'Y', or '-' for the opt-out sale", + description: "Clear - Empty", + noSaleBidders: []string{}, + ext: json.RawMessage(`{"prebid":{"nosale":["a", "b"]}}`), + expected: nil, }, { - description: "Invalid LSPA Char", - policy: Policy{Value: "1--X"}, - expectedError: "request.regs.ext.us_privacy must specify 'N', 'Y', or '-' for the limited service provider agreement", + description: "Clear - Error", + noSaleBidders: []string{}, + ext: json.RawMessage(`malformed`), + expectedError: true, }, { - description: "Invalid LSPA Case", - policy: Policy{Value: "1--y"}, - expectedError: "request.regs.ext.us_privacy must specify 'N', 'Y', or '-' for the limited service provider agreement", + description: "Write", + noSaleBidders: []string{"a", "b"}, + ext: nil, + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), + }, + { + description: "Write - Error", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`malformed`), + expectedError: true, }, } for _, test := range testCases { - result := test.policy.Validate() - - if test.expectedError == "" { - assert.NoError(t, result, test.description) - } else { - assert.EqualError(t, result, test.expectedError, test.description) - } + result, err := buildExt(test.noSaleBidders, test.ext) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, result, test.description) } } -func TestValidateConsent(t *testing.T) { +func TestBuildExtClear(t *testing.T) { testCases := []struct { description string - consent string - expectedError string + ext json.RawMessage + expected json.RawMessage + expectedError bool }{ { - description: "Valid", - consent: "1NYN", - expectedError: "", + description: "Nil Ext", + ext: nil, + expected: nil, }, { - description: "Valid - Not Applicable", - consent: "1---", - expectedError: "", + description: "Empty Ext", + ext: json.RawMessage(``), + expected: json.RawMessage(``), }, { - description: "Invalid Empty", - consent: "", - expectedError: "", + description: "Empty Ext Object", + ext: json.RawMessage(`{}`), + expected: json.RawMessage(`{}`), }, { - description: "Invalid Length", - consent: "1NY", - expectedError: "must contain 4 characters", + description: "Empty Ext.Prebid", + ext: json.RawMessage(`{"prebid":{}}`), + expected: nil, }, { - description: "Invalid Version", - consent: "2---", - expectedError: "must specify version 1", + description: "Removes Ext Entirely", + ext: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), + expected: nil, }, { - description: "Invalid Explicit Notice Char", - consent: "1X--", - expectedError: "must specify 'N', 'Y', or '-' for the explicit notice", + description: "Leaves Other Ext Values", + ext: json.RawMessage(`{"other":"any","prebid":{"nosale":["a","b"]}}`), + expected: json.RawMessage(`{"other":"any"}`), }, { - description: "Invalid Explicit Notice Case", - consent: "1y--", - expectedError: "must specify 'N', 'Y', or '-' for the explicit notice", + description: "Leaves Other Ext.Prebid Values", + ext: json.RawMessage(`{"prebid":{"nosale":["a","b"],"other":"any"}}`), + expected: json.RawMessage(`{"prebid":{"other":"any"}}`), }, { - description: "Invalid Opt-Out Sale Char", - consent: "1-X-", - expectedError: "must specify 'N', 'Y', or '-' for the opt-out sale", + description: "Leaves All Other Values", + ext: json.RawMessage(`{"other":"ABC","prebid":{"nosale":["a","b"],"other":"123"}}`), + expected: json.RawMessage(`{"other":"ABC","prebid":{"other":"123"}}`), }, { - description: "Invalid Opt-Out Sale Case", - consent: "1-y-", - expectedError: "must specify 'N', 'Y', or '-' for the opt-out sale", + description: "Malformed Ext", + ext: json.RawMessage(`malformed`), + expectedError: true, }, { - description: "Invalid LSPA Char", - consent: "1--X", - expectedError: "must specify 'N', 'Y', or '-' for the limited service provider agreement", + description: "Malformed Ext.Prebid", + ext: json.RawMessage(`{"prebid":malformed}`), + expectedError: true, }, { - description: "Invalid LSPA Case", - consent: "1--y", - expectedError: "must specify 'N', 'Y', or '-' for the limited service provider agreement", + description: "Invalid Ext.Prebid Type", + ext: json.RawMessage(`{"prebid":123}`), + expectedError: true, }, } for _, test := range testCases { - result := ValidateConsent(test.consent) - - if test.expectedError == "" { - assert.NoError(t, result, test.description) - } else { - assert.EqualError(t, result, test.expectedError, test.description) - } + result, err := buildExtClear(test.ext) + assertError(t, test.expectedError, err, test.description) + assert.Equal(t, test.expected, result, test.description) } } -func TestShouldEnforce(t *testing.T) { +func TestBuildExtWrite(t *testing.T) { testCases := []struct { - description string - policy Policy - expected bool + description string + noSaleBidders []string + ext json.RawMessage + expected json.RawMessage + expectedError bool }{ { - description: "Enforceable", - policy: Policy{Value: "1-Y-"}, - expected: true, + description: "Nil Ext", + noSaleBidders: []string{"a", "b"}, + ext: nil, + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), + }, + { + description: "Empty Ext", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(``), + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), }, { - description: "Not Enforceable - Not Present", - policy: Policy{Value: ""}, - expected: false, + description: "Empty Ext Object", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{}`), + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), }, { - description: "Not Enforceable - Opt-Out Unknown", - policy: Policy{Value: "1---"}, - expected: false, + description: "Empty Ext.Prebid", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"prebid":{}}`), + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), }, { - description: "Not Enforceable - Opt-Out Explicitly No", - policy: Policy{Value: "1-N-"}, - expected: false, + description: "Overwrites Existing", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"prebid":{"nosale":["x","y"]}}`), + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), }, { - description: "Invalid", - policy: Policy{Value: "2---"}, - expected: false, + description: "Leaves Other Ext Values", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"other":"any"}`), + expected: json.RawMessage(`{"other":"any","prebid":{"nosale":["a","b"]}}`), + }, + { + description: "Leaves Other Ext.Prebid Values", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"prebid":{"other":"any"}}`), + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"],"other":"any"}}`), + }, + { + description: "Leaves All Other Values", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"other":"ABC","prebid":{"other":"123"}}`), + expected: json.RawMessage(`{"other":"ABC","prebid":{"nosale":["a","b"],"other":"123"}}`), + }, + { + description: "Invalid Ext.Prebid No Sale Type - Still Overrides", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"prebid":{"nosale":123}}`), + expected: json.RawMessage(`{"prebid":{"nosale":["a","b"]}}`), + }, + { + description: "Invalid Ext.Prebid Type ", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"prebid":"wrongtype"}`), + expectedError: true, + }, + { + description: "Malformed Ext", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{malformed`), + expectedError: true, + }, + { + description: "Malformed Ext.Prebid", + noSaleBidders: []string{"a", "b"}, + ext: json.RawMessage(`{"prebid":malformed}`), + expectedError: true, }, } for _, test := range testCases { - result := test.policy.ShouldEnforce() + result, err := buildExtWrite(test.noSaleBidders, test.ext) + assertError(t, test.expectedError, err, test.description) assert.Equal(t, test.expected, result, test.description) } } + +func assertError(t *testing.T, expectError bool, err error, description string) { + t.Helper() + if expectError { + assert.Error(t, err, description) + } else { + assert.NoError(t, err, description) + } +} diff --git a/privacy/enforcement.go b/privacy/enforcement.go index fe81848181e..9da67bb2b15 100644 --- a/privacy/enforcement.go +++ b/privacy/enforcement.go @@ -8,14 +8,14 @@ import ( type Enforcement struct { CCPA bool COPPA bool - GDPR bool GDPRGeo bool + GDPRID bool LMT bool } // Any returns true if at least one privacy policy requires enforcement. func (e Enforcement) Any() bool { - return e.CCPA || e.COPPA || e.GDPR || e.GDPRGeo || e.LMT + return e.CCPA || e.COPPA || e.GDPRGeo || e.GDPRID || e.LMT } // Apply cleans personally identifiable information from an OpenRTB bid request. @@ -25,17 +25,33 @@ func (e Enforcement) Apply(bidRequest *openrtb.BidRequest, ampGDPRException bool func (e Enforcement) apply(bidRequest *openrtb.BidRequest, ampGDPRException bool, scrubber Scrubber) { if bidRequest != nil && e.Any() { - bidRequest.Device = scrubber.ScrubDevice(bidRequest.Device, e.getIPv6ScrubStrategy(), e.getGeoScrubStrategy()) + bidRequest.Device = scrubber.ScrubDevice(bidRequest.Device, e.getDeviceIDScrubStrategy(), e.getIPv4ScrubStrategy(), e.getIPv6ScrubStrategy(), e.getGeoScrubStrategy()) bidRequest.User = scrubber.ScrubUser(bidRequest.User, e.getUserScrubStrategy(ampGDPRException), e.getGeoScrubStrategy()) } } +func (e Enforcement) getDeviceIDScrubStrategy() ScrubStrategyDeviceID { + if e.COPPA || e.GDPRID || e.CCPA || e.LMT { + return ScrubStrategyDeviceIDAll + } + + return ScrubStrategyDeviceIDNone +} + +func (e Enforcement) getIPv4ScrubStrategy() ScrubStrategyIPV4 { + if e.COPPA || e.GDPRGeo || e.CCPA || e.LMT { + return ScrubStrategyIPV4Lowest8 + } + + return ScrubStrategyIPV4None +} + func (e Enforcement) getIPv6ScrubStrategy() ScrubStrategyIPV6 { if e.COPPA { return ScrubStrategyIPV6Lowest32 } - if e.GDPR || e.CCPA || e.LMT { + if e.GDPRGeo || e.CCPA || e.LMT { return ScrubStrategyIPV6Lowest16 } @@ -59,12 +75,11 @@ func (e Enforcement) getUserScrubStrategy(ampGDPRException bool) ScrubStrategyUs return ScrubStrategyUserIDAndDemographic } - if e.GDPR && ampGDPRException { - return ScrubStrategyUserNone + if e.CCPA || e.LMT { + return ScrubStrategyUserID } - // If no user scrubbing is needed, then return none, else scrub ID (COPPA checked above) - if e.CCPA || e.GDPR || e.LMT { + if e.GDPRID && !ampGDPRException { return ScrubStrategyUserID } diff --git a/privacy/enforcement_test.go b/privacy/enforcement_test.go index 90af24b27ea..c332f39dfd8 100644 --- a/privacy/enforcement_test.go +++ b/privacy/enforcement_test.go @@ -19,8 +19,8 @@ func TestAny(t *testing.T) { enforcement: Enforcement{ CCPA: false, COPPA: false, - GDPR: false, GDPRGeo: false, + GDPRID: false, LMT: false, }, expected: false, @@ -30,8 +30,8 @@ func TestAny(t *testing.T) { enforcement: Enforcement{ CCPA: true, COPPA: true, - GDPR: true, GDPRGeo: true, + GDPRID: true, LMT: true, }, expected: true, @@ -41,8 +41,8 @@ func TestAny(t *testing.T) { enforcement: Enforcement{ CCPA: false, COPPA: true, - GDPR: false, GDPRGeo: false, + GDPRID: false, LMT: true, }, expected: true, @@ -60,6 +60,8 @@ func TestApply(t *testing.T) { description string enforcement Enforcement ampGDPRException bool + expectedDeviceID ScrubStrategyDeviceID + expectedDeviceIPv4 ScrubStrategyIPV4 expectedDeviceIPv6 ScrubStrategyIPV6 expectedDeviceGeo ScrubStrategyGeo expectedUser ScrubStrategyUser @@ -70,11 +72,12 @@ func TestApply(t *testing.T) { enforcement: Enforcement{ CCPA: true, COPPA: true, - GDPR: true, GDPRGeo: true, + GDPRID: true, LMT: true, }, - ampGDPRException: false, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, expectedDeviceIPv6: ScrubStrategyIPV6Lowest32, expectedDeviceGeo: ScrubStrategyGeoFull, expectedUser: ScrubStrategyUserIDAndDemographic, @@ -85,11 +88,12 @@ func TestApply(t *testing.T) { enforcement: Enforcement{ CCPA: true, COPPA: false, - GDPR: false, GDPRGeo: false, + GDPRID: false, LMT: false, }, - ampGDPRException: false, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, expectedUser: ScrubStrategyUserID, @@ -100,102 +104,98 @@ func TestApply(t *testing.T) { enforcement: Enforcement{ CCPA: false, COPPA: true, - GDPR: false, GDPRGeo: false, + GDPRID: false, LMT: false, }, - ampGDPRException: false, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, expectedDeviceIPv6: ScrubStrategyIPV6Lowest32, expectedDeviceGeo: ScrubStrategyGeoFull, expectedUser: ScrubStrategyUserIDAndDemographic, expectedUserGeo: ScrubStrategyGeoFull, }, { - description: "GDPR Only", + description: "GDPR Only - Full", enforcement: Enforcement{ CCPA: false, COPPA: false, - GDPR: true, GDPRGeo: true, + GDPRID: true, LMT: false, }, ampGDPRException: false, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, expectedUser: ScrubStrategyUserID, expectedUserGeo: ScrubStrategyGeoReducedPrecision, }, { - description: "GDPR Only, ampGDPRException", + description: "GDPR Only - Full - AMP Exception", enforcement: Enforcement{ CCPA: false, COPPA: false, - GDPR: true, GDPRGeo: true, + GDPRID: true, LMT: false, }, ampGDPRException: true, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, expectedUser: ScrubStrategyUserNone, expectedUserGeo: ScrubStrategyGeoReducedPrecision, }, { - description: "CCPA Only, ampGDPRException", + description: "GDPR Only - ID Only", enforcement: Enforcement{ - CCPA: true, + CCPA: false, COPPA: false, - GDPR: false, GDPRGeo: false, + GDPRID: true, LMT: false, }, - ampGDPRException: true, - expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, - expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, + ampGDPRException: false, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4None, + expectedDeviceIPv6: ScrubStrategyIPV6None, + expectedDeviceGeo: ScrubStrategyGeoNone, expectedUser: ScrubStrategyUserID, - expectedUserGeo: ScrubStrategyGeoReducedPrecision, - }, - { - description: "COPPA and GDPR, ampGDPRException", - enforcement: Enforcement{ - CCPA: false, - COPPA: true, - GDPR: true, - GDPRGeo: true, - LMT: false, - }, - ampGDPRException: true, - expectedDeviceIPv6: ScrubStrategyIPV6Lowest32, - expectedDeviceGeo: ScrubStrategyGeoFull, - expectedUser: ScrubStrategyUserIDAndDemographic, - expectedUserGeo: ScrubStrategyGeoFull, + expectedUserGeo: ScrubStrategyGeoNone, }, { - description: "GDPR Only, no Geo", + description: "GDPR Only - ID Only - AMP Exception", enforcement: Enforcement{ CCPA: false, COPPA: false, - GDPR: true, GDPRGeo: false, + GDPRID: true, LMT: false, }, - ampGDPRException: false, - expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, + ampGDPRException: true, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4None, + expectedDeviceIPv6: ScrubStrategyIPV6None, expectedDeviceGeo: ScrubStrategyGeoNone, - expectedUser: ScrubStrategyUserID, + expectedUser: ScrubStrategyUserNone, expectedUserGeo: ScrubStrategyGeoNone, }, { - description: "GDPR Only, Geo only", + description: "GDPR Only - Geo Only", enforcement: Enforcement{ CCPA: false, COPPA: false, - GDPR: false, GDPRGeo: true, + GDPRID: false, LMT: false, }, ampGDPRException: false, - expectedDeviceIPv6: ScrubStrategyIPV6None, + expectedDeviceID: ScrubStrategyDeviceIDNone, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, + expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, expectedUser: ScrubStrategyUserNone, expectedUserGeo: ScrubStrategyGeoReducedPrecision, @@ -205,30 +205,50 @@ func TestApply(t *testing.T) { enforcement: Enforcement{ CCPA: false, COPPA: false, - GDPR: false, GDPRGeo: false, + GDPRID: false, LMT: true, }, - ampGDPRException: false, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, expectedUser: ScrubStrategyUserID, expectedUserGeo: ScrubStrategyGeoReducedPrecision, }, { - description: "LMT Only, ampGDPRException", + description: "Interactions: COPPA Only + AMP Exception", enforcement: Enforcement{ CCPA: false, - COPPA: false, - GDPR: false, + COPPA: true, GDPRGeo: false, - LMT: true, + GDPRID: false, + LMT: false, }, ampGDPRException: true, - expectedDeviceIPv6: ScrubStrategyIPV6Lowest16, - expectedDeviceGeo: ScrubStrategyGeoReducedPrecision, - expectedUser: ScrubStrategyUserID, - expectedUserGeo: ScrubStrategyGeoReducedPrecision, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, + expectedDeviceIPv6: ScrubStrategyIPV6Lowest32, + expectedDeviceGeo: ScrubStrategyGeoFull, + expectedUser: ScrubStrategyUserIDAndDemographic, + expectedUserGeo: ScrubStrategyGeoFull, + }, + { + description: "Interactions: COPPA + GDPR Full + AMP Exception", + enforcement: Enforcement{ + CCPA: false, + COPPA: true, + GDPRGeo: true, + GDPRID: true, + LMT: false, + }, + ampGDPRException: true, + expectedDeviceID: ScrubStrategyDeviceIDAll, + expectedDeviceIPv4: ScrubStrategyIPV4Lowest8, + expectedDeviceIPv6: ScrubStrategyIPV6Lowest32, + expectedDeviceGeo: ScrubStrategyGeoFull, + expectedUser: ScrubStrategyUserIDAndDemographic, + expectedUserGeo: ScrubStrategyGeoFull, }, } @@ -241,7 +261,7 @@ func TestApply(t *testing.T) { replacedUser := &openrtb.User{} m := &mockScrubber{} - m.On("ScrubDevice", req.Device, test.expectedDeviceIPv6, test.expectedDeviceGeo).Return(replacedDevice).Once() + m.On("ScrubDevice", req.Device, test.expectedDeviceID, test.expectedDeviceIPv4, test.expectedDeviceIPv6, test.expectedDeviceGeo).Return(replacedDevice).Once() m.On("ScrubUser", req.User, test.expectedUser, test.expectedUserGeo).Return(replacedUser).Once() test.enforcement.apply(req, test.ampGDPRException, m) @@ -258,10 +278,11 @@ func TestApplyNoneApplicable(t *testing.T) { m := &mockScrubber{} enforcement := Enforcement{ - CCPA: false, - COPPA: false, - GDPR: false, - LMT: false, + CCPA: false, + COPPA: false, + GDPRGeo: false, + GDPRID: false, + LMT: false, } enforcement.apply(req, false, m) @@ -283,8 +304,8 @@ type mockScrubber struct { mock.Mock } -func (m *mockScrubber) ScrubDevice(device *openrtb.Device, ipv6 ScrubStrategyIPV6, geo ScrubStrategyGeo) *openrtb.Device { - args := m.Called(device, ipv6, geo) +func (m *mockScrubber) ScrubDevice(device *openrtb.Device, id ScrubStrategyDeviceID, ipv4 ScrubStrategyIPV4, ipv6 ScrubStrategyIPV6, geo ScrubStrategyGeo) *openrtb.Device { + args := m.Called(device, id, ipv4, ipv6, geo) return args.Get(0).(*openrtb.Device) } diff --git a/privacy/enforcer.go b/privacy/enforcer.go new file mode 100644 index 00000000000..0d5ecad5309 --- /dev/null +++ b/privacy/enforcer.go @@ -0,0 +1,43 @@ +package privacy + +// PolicyEnforcer determines if personally identifiable information (PII) should be removed or anonymized per the policy. +type PolicyEnforcer interface { + // CanEnforce returns true when policy information is specifically provided by the publisher. + CanEnforce() bool + + // ShouldEnforce returns true when the OpenRTB request should have personally identifiable + // information (PII) removed or anonymized per the policy. + ShouldEnforce(bidder string) bool +} + +// NilPolicyEnforcer implements the PolicyEnforcer interface but will always return false. +type NilPolicyEnforcer struct{} + +// CanEnforce is hardcoded to always return false. +func (NilPolicyEnforcer) CanEnforce() bool { + return false +} + +// ShouldEnforce is hardcoded to always return false. +func (NilPolicyEnforcer) ShouldEnforce(bidder string) bool { + return false +} + +// EnabledPolicyEnforcer decorates a PolicyEnforcer with an enabled flag. +type EnabledPolicyEnforcer struct { + Enabled bool + PolicyEnforcer PolicyEnforcer +} + +// CanEnforce returns true when the PolicyEnforcer can enforce. +func (p EnabledPolicyEnforcer) CanEnforce() bool { + return p.PolicyEnforcer.CanEnforce() +} + +// ShouldEnforce returns true when the enforcer is enabled the PolicyEnforcer allows enforcement. +func (p EnabledPolicyEnforcer) ShouldEnforce(bidder string) bool { + if p.Enabled { + return p.PolicyEnforcer.ShouldEnforce(bidder) + } + return false +} diff --git a/privacy/enforcer_test.go b/privacy/enforcer_test.go new file mode 100644 index 00000000000..b0c4032c714 --- /dev/null +++ b/privacy/enforcer_test.go @@ -0,0 +1,18 @@ +package privacy + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestNilEnforcerCanEnforce(t *testing.T) { + nilEnforcer := &NilPolicyEnforcer{} + assert.False(t, nilEnforcer.CanEnforce()) +} + +func TestNilEnforcerShouldEnforce(t *testing.T) { + nilEnforcer := &NilPolicyEnforcer{} + assert.False(t, nilEnforcer.ShouldEnforce("")) + assert.False(t, nilEnforcer.ShouldEnforce("anyBidder")) +} diff --git a/privacy/gdpr/consentwriter.go b/privacy/gdpr/consentwriter.go new file mode 100644 index 00000000000..f1cc2ce12f7 --- /dev/null +++ b/privacy/gdpr/consentwriter.go @@ -0,0 +1,44 @@ +package gdpr + +import ( + "encoding/json" + + "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" + + "github.com/PubMatic-OpenWrap/openrtb" +) + +// ConsentWriter implements the PolicyWriter interface for GDPR TCF. +type ConsentWriter struct { + Consent string +} + +// Write mutates an OpenRTB bid request with the GDPR TCF consent. +func (c ConsentWriter) Write(req *openrtb.BidRequest) error { + if c.Consent == "" { + return nil + } + + if req.User == nil { + req.User = &openrtb.User{} + } + + if req.User.Ext == nil { + ext, err := json.Marshal(openrtb_ext.ExtUser{Consent: c.Consent}) + if err == nil { + req.User.Ext = ext + } + return err + } + + var extMap map[string]interface{} + err := json.Unmarshal(req.User.Ext, &extMap) + if err == nil { + extMap["consent"] = c.Consent + ext, err := json.Marshal(extMap) + if err == nil { + req.User.Ext = ext + } + } + return err +} diff --git a/privacy/gdpr/consentwriter_test.go b/privacy/gdpr/consentwriter_test.go new file mode 100644 index 00000000000..65df8051d02 --- /dev/null +++ b/privacy/gdpr/consentwriter_test.go @@ -0,0 +1,101 @@ +package gdpr + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/stretchr/testify/assert" +) + +func TestConsentWriter(t *testing.T) { + testCases := []struct { + description string + consent string + request *openrtb.BidRequest + expected *openrtb.BidRequest + expectedError bool + }{ + { + description: "Empty", + consent: "", + request: &openrtb.BidRequest{}, + expected: &openrtb.BidRequest{}, + }, + { + description: "Enabled With Nil Request User Object", + consent: "anyConsent", + request: &openrtb.BidRequest{}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"anyConsent"}`)}}, + }, + { + description: "Enabled With Nil Request User Ext Object", + consent: "anyConsent", + request: &openrtb.BidRequest{User: &openrtb.User{}}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"anyConsent"}`)}}, + }, + { + description: "Enabled With Existing Request User Ext Object - Doesn't Overwrite", + consent: "anyConsent", + request: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"existing":"any"}`)}}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"anyConsent","existing":"any"}`)}}, + }, + { + description: "Enabled With Existing Request User Ext Object - Overwrites", + consent: "anyConsent", + request: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"existing":"any","consent":"toBeOverwritten"}`)}}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"anyConsent","existing":"any"}`)}}, + }, + { + description: "Enabled With Existing Malformed Request User Ext Object", + consent: "anyConsent", + request: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`malformed`)}}, + expectedError: true, + }, + { + description: "Injection Attack With Nil Request User Object", + consent: "BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"", + request: &openrtb.BidRequest{}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), + }}, + }, + { + description: "Injection Attack With Nil Request User Ext Object", + consent: "BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"", + request: &openrtb.BidRequest{User: &openrtb.User{}}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), + }}, + }, + { + description: "Injection Attack With Existing Request User Ext Object", + consent: "BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"", + request: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"existing":"any"}`), + }}, + expected: &openrtb.BidRequest{User: &openrtb.User{ + Ext: json.RawMessage(`{"consent":"BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"","existing":"any"}`), + }}, + }, + } + + for _, test := range testCases { + writer := ConsentWriter{test.consent} + err := writer.Write(test.request) + + if test.expectedError { + assert.Error(t, err, test.description) + } else { + assert.NoError(t, err, test.description) + assert.Equal(t, test.expected, test.request, test.description) + } + } +} diff --git a/privacy/gdpr/policy.go b/privacy/gdpr/policy.go index 9c910b5e6f2..0464a9ff979 100644 --- a/privacy/gdpr/policy.go +++ b/privacy/gdpr/policy.go @@ -1,10 +1,6 @@ package gdpr import ( - "encoding/json" - "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" - - "github.com/PubMatic-OpenWrap/openrtb" "github.com/prebid/go-gdpr/vendorconsent" ) @@ -14,38 +10,8 @@ type Policy struct { Consent string } -// Write mutates an OpenRTB bid request with the context of the GDPR policy. -func (p Policy) Write(req *openrtb.BidRequest) error { - if p.Consent == "" { - return nil - } - - if req.User == nil { - req.User = &openrtb.User{} - } - - if req.User.Ext == nil { - ext, err := json.Marshal(openrtb_ext.ExtUser{Consent: p.Consent}) - if err == nil { - req.User.Ext = ext - } - return err - } - - var extMap map[string]interface{} - err := json.Unmarshal(req.User.Ext, &extMap) - if err == nil { - extMap["consent"] = p.Consent - ext, err := json.Marshal(extMap) - if err == nil { - req.User.Ext = ext - } - } - return err -} - -// ValidateConsent returns an error if the GDPR consent string does not adhere to the IAB TCF spec. -func ValidateConsent(consent string) error { +// ValidateConsent returns true if the consent string is empty or valid per the IAB TCF spec. +func ValidateConsent(consent string) bool { _, err := vendorconsent.ParseString(consent) - return err + return err == nil } diff --git a/privacy/gdpr/policy_test.go b/privacy/gdpr/policy_test.go index ff1b8827a2f..dc8f56425c5 100644 --- a/privacy/gdpr/policy_test.go +++ b/privacy/gdpr/policy_test.go @@ -1,129 +1,36 @@ package gdpr import ( - "encoding/json" "testing" - "github.com/PubMatic-OpenWrap/openrtb" "github.com/stretchr/testify/assert" ) -func TestWrite(t *testing.T) { - testCases := []struct { - description string - policy Policy - request *openrtb.BidRequest - expected *openrtb.BidRequest - expectedError bool - }{ - { - description: "Disabled", - policy: Policy{Consent: ""}, - request: &openrtb.BidRequest{}, - expected: &openrtb.BidRequest{}, - }, - { - description: "Enabled With Nil Request User Object", - policy: Policy{Consent: "anyConsent"}, - request: &openrtb.BidRequest{}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"anyConsent"}`)}}, - }, - { - description: "Enabled With Nil Request User Ext Object", - policy: Policy{Consent: "anyConsent"}, - request: &openrtb.BidRequest{User: &openrtb.User{}}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"anyConsent"}`)}}, - }, - { - description: "Enabled With Existing Request User Ext Object - Doesn't Overwrite", - policy: Policy{Consent: "anyConsent"}, - request: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"existing":"any"}`)}}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"anyConsent","existing":"any"}`)}}, - }, - { - description: "Enabled With Existing Request User Ext Object - Overwrites", - policy: Policy{Consent: "anyConsent"}, - request: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"existing":"any","consent":"toBeOverwritten"}`)}}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"anyConsent","existing":"any"}`)}}, - }, - { - description: "Enabled With Existing Malformed Request User Ext Object", - policy: Policy{Consent: "anyConsent"}, - request: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`malformed`)}}, - expectedError: true, - }, - { - description: "Injection Attack With Nil Request User Object", - policy: Policy{Consent: "BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}, - request: &openrtb.BidRequest{}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), - }}, - }, - { - description: "Injection Attack With Nil Request User Ext Object", - policy: Policy{Consent: "BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}, - request: &openrtb.BidRequest{User: &openrtb.User{}}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}`), - }}, - }, - { - description: "Injection Attack With Existing Request User Ext Object", - policy: Policy{Consent: "BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\""}, - request: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"existing":"any"}`), - }}, - expected: &openrtb.BidRequest{User: &openrtb.User{ - Ext: json.RawMessage(`{"consent":"BONV8oqONXwgmADACHENAO7pqzAAppY\"},\"oops\":\"malicious\",\"p\":{\"p\":\"","existing":"any"}`), - }}, - }, - } - - for _, test := range testCases { - err := test.policy.Write(test.request) - - if test.expectedError { - assert.Error(t, err, test.description) - } else { - assert.NoError(t, err, test.description) - assert.Equal(t, test.expected, test.request, test.description) - } - } -} - func TestValidateConsent(t *testing.T) { testCases := []struct { description string consent string - expectError bool + expected bool }{ { description: "Invalid", consent: "", - expectError: true, + expected: false, }, { - description: "Valid", + description: "TCF1 Valid", consent: "BONV8oqONXwgmADACHENAO7pqzAAppY", - expectError: false, + expected: true, + }, + { + description: "TCF2 Valid", + consent: "COzTVhaOzTVhaGvAAAENAiCIAP_AAH_AAAAAAEEUACCKAAA", + expected: true, }, } for _, test := range testCases { result := ValidateConsent(test.consent) - - if test.expectError { - assert.Error(t, result, test.description) - } else { - assert.NoError(t, result, test.description) - } + assert.Equal(t, test.expected, result, test.description) } } diff --git a/privacy/lmt/policy.go b/privacy/lmt/policy.go index bdbc1a2b34b..5f23b9a3eef 100644 --- a/privacy/lmt/policy.go +++ b/privacy/lmt/policy.go @@ -15,19 +15,21 @@ type Policy struct { SignalProvided bool } -// ReadPolicy extracts the LMT (Limit Ad Tracking) policy from an OpenRTB bid request. -func ReadPolicy(req *openrtb.BidRequest) Policy { - policy := Policy{} - +// ReadFromRequest extracts the LMT (Limit Ad Tracking) policy from an OpenRTB bid request. +func ReadFromRequest(req *openrtb.BidRequest) (policy Policy) { if req != nil && req.Device != nil && req.Device.Lmt != nil { policy.Signal = int(*req.Device.Lmt) policy.SignalProvided = true } + return +} - return policy +// CanEnforce returns true the LMT (Limit Ad Tracking) signal is provided by the publisher. +func (p Policy) CanEnforce() bool { + return p.SignalProvided } // ShouldEnforce returns true when the LMT (Limit Ad Tracking) policy is in effect. -func (p Policy) ShouldEnforce() bool { +func (p Policy) ShouldEnforce(bidder string) bool { return p.SignalProvided && p.Signal == trackingRestricted } diff --git a/privacy/lmt/policy_test.go b/privacy/lmt/policy_test.go index 12ea1870d2f..9d0e3b6aa9a 100644 --- a/privacy/lmt/policy_test.go +++ b/privacy/lmt/policy_test.go @@ -7,7 +7,7 @@ import ( "github.com/stretchr/testify/assert" ) -func TestRead(t *testing.T) { +func TestReadFromRequest(t *testing.T) { var one int8 = 1 testCases := []struct { @@ -60,11 +60,73 @@ func TestRead(t *testing.T) { } for _, test := range testCases { - p := ReadPolicy(test.request) + p := ReadFromRequest(test.request) assert.Equal(t, test.expectedPolicy, p, test.description) } } +func TestCanEnforce(t *testing.T) { + testCases := []struct { + description string + policy Policy + expected bool + }{ + { + description: "Signal Not Provided - Zero", + policy: Policy{ + Signal: 0, + SignalProvided: false, + }, + expected: false, + }, + { + description: "Signal Not Provided - One", + policy: Policy{ + Signal: 1, + SignalProvided: false, + }, + expected: false, + }, + { + description: "Signal Not Provided - Other", + policy: Policy{ + Signal: 42, + SignalProvided: false, + }, + expected: false, + }, + { + description: "Signal Provided - Zero", + policy: Policy{ + Signal: 0, + SignalProvided: true, + }, + expected: true, + }, + { + description: "Signal Provided - One", + policy: Policy{ + Signal: 1, + SignalProvided: true, + }, + expected: true, + }, + { + description: "Signal Provided - Other", + policy: Policy{ + Signal: 42, + SignalProvided: true, + }, + expected: true, + }, + } + + for _, test := range testCases { + result := test.policy.CanEnforce() + assert.Equal(t, test.expected, result, test.description) + } +} + func TestShouldEnforce(t *testing.T) { testCases := []struct { description string @@ -122,7 +184,7 @@ func TestShouldEnforce(t *testing.T) { } for _, test := range testCases { - result := test.policy.ShouldEnforce() + result := test.policy.ShouldEnforce("") assert.Equal(t, test.expected, result, test.description) } } diff --git a/privacy/policies.go b/privacy/policies.go index 837d2fa05c3..a1c3fca49be 100644 --- a/privacy/policies.go +++ b/privacy/policies.go @@ -1,60 +1,14 @@ package privacy import ( - "github.com/PubMatic-OpenWrap/openrtb" - "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" + "github.com/PubMatic-OpenWrap/prebid-server/privacy/lmt" ) // Policies represents the privacy regulations for an OpenRTB bid request. type Policies struct { - GDPR gdpr.Policy CCPA ccpa.Policy -} - -type policyWriter interface { - Write(req *openrtb.BidRequest) error -} - -// Write mutates an OpenRTB bid request with the policies applied. -func (p Policies) Write(req *openrtb.BidRequest) error { - return writePolicies(req, []policyWriter{ - p.GDPR, p.CCPA, - }) -} - -func writePolicies(req *openrtb.BidRequest, writers []policyWriter) error { - for _, writer := range writers { - if err := writer.Write(req); err != nil { - return err - } - } - - return nil -} - -// ReadPoliciesFromConsent inspects the consent string kind and sets the corresponding values in a new Policies object. -func ReadPoliciesFromConsent(consent string) (Policies, bool) { - if len(consent) == 0 { - return Policies{}, false - } - - if err := gdpr.ValidateConsent(consent); err == nil { - return Policies{ - GDPR: gdpr.Policy{ - Consent: consent, - }, - }, true - } - - if err := ccpa.ValidateConsent(consent); err == nil { - return Policies{ - CCPA: ccpa.Policy{ - Value: consent, - }, - }, true - } - - return Policies{}, false + GDPR gdpr.Policy + LMT lmt.Policy } diff --git a/privacy/policies_test.go b/privacy/policies_test.go deleted file mode 100644 index a7650193892..00000000000 --- a/privacy/policies_test.go +++ /dev/null @@ -1,119 +0,0 @@ -package privacy - -import ( - "errors" - "testing" - - "github.com/PubMatic-OpenWrap/openrtb" - "github.com/PubMatic-OpenWrap/prebid-server/privacy/ccpa" - "github.com/PubMatic-OpenWrap/prebid-server/privacy/gdpr" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/mock" -) - -func TestWritePoliciesNone(t *testing.T) { - request := &openrtb.BidRequest{} - policyWriters := []policyWriter{} - - err := writePolicies(request, policyWriters) - - assert.NoError(t, err) -} - -func TestWritePoliciesOne(t *testing.T) { - request := &openrtb.BidRequest{} - mockWriter := new(mockPolicyWriter) - policyWriters := []policyWriter{ - mockWriter, - } - - mockWriter.On("Write", request).Return(nil).Once() - - err := writePolicies(request, policyWriters) - - assert.NoError(t, err) - mockWriter.AssertExpectations(t) -} - -func TestWritePoliciesMany(t *testing.T) { - request := &openrtb.BidRequest{} - mockWriter1 := new(mockPolicyWriter) - mockWriter2 := new(mockPolicyWriter) - policyWriters := []policyWriter{ - mockWriter1, mockWriter2, - } - - mockWriter1.On("Write", request).Return(nil).Once() - mockWriter2.On("Write", request).Return(nil).Once() - - err := writePolicies(request, policyWriters) - - assert.NoError(t, err) - mockWriter1.AssertExpectations(t) - mockWriter2.AssertExpectations(t) -} - -func TestWritePoliciesError(t *testing.T) { - request := &openrtb.BidRequest{} - mockWriter := new(mockPolicyWriter) - policyWriters := []policyWriter{ - mockWriter, - } - - expectedErr := errors.New("anyError") - mockWriter.On("Write", request).Return(expectedErr).Once() - - err := writePolicies(request, policyWriters) - - assert.Error(t, err, expectedErr) - mockWriter.AssertExpectations(t) -} - -type mockPolicyWriter struct { - mock.Mock -} - -func (m *mockPolicyWriter) Write(req *openrtb.BidRequest) error { - args := m.Called(req) - return args.Error(0) -} - -func TestReadPoliciesFromConsent(t *testing.T) { - testCases := []struct { - description string - consent string - expectedResultValue Policies - expectedResultOK bool - }{ - { - description: "Empty String", - consent: "", - expectedResultValue: Policies{}, - expectedResultOK: false, - }, - { - description: "CCPA", - consent: "1NYN", - expectedResultValue: Policies{CCPA: ccpa.Policy{Value: "1NYN"}}, - expectedResultOK: true, - }, - { - description: "GDPR TCF 1.0", - consent: "BONV8oqONXwgmADACHENAO7pqzAAppY", - expectedResultValue: Policies{GDPR: gdpr.Policy{Consent: "BONV8oqONXwgmADACHENAO7pqzAAppY"}}, - expectedResultOK: true, - }, - { - description: "Invalid", - consent: "any invalid", - expectedResultValue: Policies{}, - expectedResultOK: false, - }, - } - - for _, test := range testCases { - resultValue, resultOK := ReadPoliciesFromConsent(test.consent) - assert.Equal(t, test.expectedResultValue, resultValue, test.description+":value") - assert.Equal(t, test.expectedResultOK, resultOK, test.description+":ok") - } -} diff --git a/privacy/scrubber.go b/privacy/scrubber.go index 0bb1029faf5..aea5c9008f4 100644 --- a/privacy/scrubber.go +++ b/privacy/scrubber.go @@ -7,6 +7,17 @@ import ( "github.com/PubMatic-OpenWrap/openrtb" ) +// ScrubStrategyIPV4 defines the approach to scrub PII from an IPV4 address. +type ScrubStrategyIPV4 int + +const ( + // ScrubStrategyIPV4None does not remove any part of an IPV4 address. + ScrubStrategyIPV4None ScrubStrategyIPV4 = iota + + // ScrubStrategyIPV4Lowest8 zeroes out the last 8 bits of an IPV4 address. + ScrubStrategyIPV4Lowest8 +) + // ScrubStrategyIPV6 defines the approach to scrub PII from an IPV6 address. type ScrubStrategyIPV6 int @@ -49,9 +60,20 @@ const ( ScrubStrategyUserID ) +// ScrubStrategyDeviceID defines the approach to remove hardware id and device id data. +type ScrubStrategyDeviceID int + +const ( + // ScrubStrategyDeviceIDNone does not remove hardware id and device id data. + ScrubStrategyDeviceIDNone ScrubStrategyDeviceID = iota + + // ScrubStrategyDeviceIDAll removes all hardware and device id data (ifa, mac hashes device id hashes) + ScrubStrategyDeviceIDAll +) + // Scrubber removes PII from parts of an OpenRTB request. type Scrubber interface { - ScrubDevice(device *openrtb.Device, ipv6 ScrubStrategyIPV6, geo ScrubStrategyGeo) *openrtb.Device + ScrubDevice(device *openrtb.Device, id ScrubStrategyDeviceID, ipv4 ScrubStrategyIPV4, ipv6 ScrubStrategyIPV6, geo ScrubStrategyGeo) *openrtb.Device ScrubUser(user *openrtb.User, strategy ScrubStrategyUser, geo ScrubStrategyGeo) *openrtb.User } @@ -62,20 +84,28 @@ func NewScrubber() Scrubber { return scrubber{} } -func (scrubber) ScrubDevice(device *openrtb.Device, ipv6 ScrubStrategyIPV6, geo ScrubStrategyGeo) *openrtb.Device { +func (scrubber) ScrubDevice(device *openrtb.Device, id ScrubStrategyDeviceID, ipv4 ScrubStrategyIPV4, ipv6 ScrubStrategyIPV6, geo ScrubStrategyGeo) *openrtb.Device { if device == nil { return nil } deviceCopy := *device - deviceCopy.DIDMD5 = "" - deviceCopy.DIDSHA1 = "" - deviceCopy.DPIDMD5 = "" - deviceCopy.DPIDSHA1 = "" - deviceCopy.IFA = "" - deviceCopy.MACMD5 = "" - deviceCopy.MACSHA1 = "" - deviceCopy.IP = scrubIPV4(device.IP) + + switch id { + case ScrubStrategyDeviceIDAll: + deviceCopy.DIDMD5 = "" + deviceCopy.DIDSHA1 = "" + deviceCopy.DPIDMD5 = "" + deviceCopy.DPIDSHA1 = "" + deviceCopy.IFA = "" + deviceCopy.MACMD5 = "" + deviceCopy.MACSHA1 = "" + } + + switch ipv4 { + case ScrubStrategyIPV4Lowest8: + deviceCopy.IP = scrubIPV4Lowest8(device.IP) + } switch ipv6 { case ScrubStrategyIPV6Lowest16: @@ -124,7 +154,7 @@ func (scrubber) ScrubUser(user *openrtb.User, strategy ScrubStrategyUser, geo Sc return &userCopy } -func scrubIPV4(ip string) string { +func scrubIPV4Lowest8(ip string) string { i := strings.LastIndex(ip, ".") if i == -1 { return "" diff --git a/privacy/scrubber_test.go b/privacy/scrubber_test.go index f33bb5fd996..4d989e1c5a1 100644 --- a/privacy/scrubber_test.go +++ b/privacy/scrubber_test.go @@ -31,28 +31,21 @@ func TestScrubDevice(t *testing.T) { testCases := []struct { description string expected *openrtb.Device + id ScrubStrategyDeviceID + ipv4 ScrubStrategyIPV4 ipv6 ScrubStrategyIPV6 geo ScrubStrategyGeo }{ { - description: "IPv6 Lowest 32 & Geo Full", - expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", - IP: "1.2.3.0", - IPv6: "2001:0db8:0000:0000:0000:ff00:0:0", - Geo: &openrtb.Geo{}, - }, - ipv6: ScrubStrategyIPV6Lowest32, - geo: ScrubStrategyGeoFull, + description: "All Strageties - None", + expected: device, + id: ScrubStrategyDeviceIDNone, + ipv4: ScrubStrategyIPV4None, + ipv6: ScrubStrategyIPV6None, + geo: ScrubStrategyGeoNone, }, { - description: "IPv6 Lowest 16 & Geo Full", + description: "All Strageties - Strictest", expected: &openrtb.Device{ DIDMD5: "", DIDSHA1: "", @@ -62,14 +55,16 @@ func TestScrubDevice(t *testing.T) { MACMD5: "", IFA: "", IP: "1.2.3.0", - IPv6: "2001:0db8:0000:0000:0000:ff00:0042:0", + IPv6: "2001:0db8:0000:0000:0000:ff00:0:0", Geo: &openrtb.Geo{}, }, - ipv6: ScrubStrategyIPV6Lowest16, + id: ScrubStrategyDeviceIDAll, + ipv4: ScrubStrategyIPV4Lowest8, + ipv6: ScrubStrategyIPV6Lowest32, geo: ScrubStrategyGeoFull, }, { - description: "IPv6 None & Geo Full", + description: "Isolated - ID - All", expected: &openrtb.Device{ DIDMD5: "", DIDSHA1: "", @@ -78,161 +73,126 @@ func TestScrubDevice(t *testing.T) { MACSHA1: "", MACMD5: "", IFA: "", - IP: "1.2.3.0", + IP: "1.2.3.4", IPv6: "2001:0db8:0000:0000:0000:ff00:0042:8329", - Geo: &openrtb.Geo{}, + Geo: device.Geo, }, + id: ScrubStrategyDeviceIDAll, + ipv4: ScrubStrategyIPV4None, ipv6: ScrubStrategyIPV6None, - geo: ScrubStrategyGeoFull, + geo: ScrubStrategyGeoNone, }, { - description: "IPv6 Lowest 32 & Geo Reduced", + description: "Isolated - IPv4 - Lowest 8", expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", + DIDMD5: "anyDIDMD5", + DIDSHA1: "anyDIDSHA1", + DPIDMD5: "anyDPIDMD5", + DPIDSHA1: "anyDPIDSHA1", + MACSHA1: "anyMACSHA1", + MACMD5: "anyMACMD5", + IFA: "anyIFA", IP: "1.2.3.0", - IPv6: "2001:0db8:0000:0000:0000:ff00:0:0", - Geo: &openrtb.Geo{ - Lat: 123.46, - Lon: 678.89, - Metro: "some metro", - City: "some city", - ZIP: "some zip", - }, + IPv6: "2001:0db8:0000:0000:0000:ff00:0042:8329", + Geo: device.Geo, }, - ipv6: ScrubStrategyIPV6Lowest32, - geo: ScrubStrategyGeoReducedPrecision, + id: ScrubStrategyDeviceIDNone, + ipv4: ScrubStrategyIPV4Lowest8, + ipv6: ScrubStrategyIPV6None, + geo: ScrubStrategyGeoNone, }, { - description: "IPv6 Lowest 16 & Geo Reduced", + description: "Isolated - IPv6 - Lowest 16", expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", - IP: "1.2.3.0", + DIDMD5: "anyDIDMD5", + DIDSHA1: "anyDIDSHA1", + DPIDMD5: "anyDPIDMD5", + DPIDSHA1: "anyDPIDSHA1", + MACSHA1: "anyMACSHA1", + MACMD5: "anyMACMD5", + IFA: "anyIFA", + IP: "1.2.3.4", IPv6: "2001:0db8:0000:0000:0000:ff00:0042:0", - Geo: &openrtb.Geo{ - Lat: 123.46, - Lon: 678.89, - Metro: "some metro", - City: "some city", - ZIP: "some zip", - }, + Geo: device.Geo, }, + id: ScrubStrategyDeviceIDNone, + ipv4: ScrubStrategyIPV4None, ipv6: ScrubStrategyIPV6Lowest16, - geo: ScrubStrategyGeoReducedPrecision, - }, - { - description: "IPv6 None & Geo Reduced", - expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", - IP: "1.2.3.0", - IPv6: "2001:0db8:0000:0000:0000:ff00:0042:8329", - Geo: &openrtb.Geo{ - Lat: 123.46, - Lon: 678.89, - Metro: "some metro", - City: "some city", - ZIP: "some zip", - }, - }, - ipv6: ScrubStrategyIPV6None, - geo: ScrubStrategyGeoReducedPrecision, + geo: ScrubStrategyGeoNone, }, { - description: "IPv6 Lowest 32 & Geo None", + description: "Isolated - IPv6 - Lowest 32", expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", - IP: "1.2.3.0", + DIDMD5: "anyDIDMD5", + DIDSHA1: "anyDIDSHA1", + DPIDMD5: "anyDPIDMD5", + DPIDSHA1: "anyDPIDSHA1", + MACSHA1: "anyMACSHA1", + MACMD5: "anyMACMD5", + IFA: "anyIFA", + IP: "1.2.3.4", IPv6: "2001:0db8:0000:0000:0000:ff00:0:0", - Geo: &openrtb.Geo{ - Lat: 123.456, - Lon: 678.89, - Metro: "some metro", - City: "some city", - ZIP: "some zip", - }, + Geo: device.Geo, }, + id: ScrubStrategyDeviceIDNone, + ipv4: ScrubStrategyIPV4None, ipv6: ScrubStrategyIPV6Lowest32, geo: ScrubStrategyGeoNone, }, { - description: "IPv6 Lowest 16 & Geo None", + description: "Isolated - Geo - Reduced Precision", expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", - IP: "1.2.3.0", - IPv6: "2001:0db8:0000:0000:0000:ff00:0042:0", + DIDMD5: "anyDIDMD5", + DIDSHA1: "anyDIDSHA1", + DPIDMD5: "anyDPIDMD5", + DPIDSHA1: "anyDPIDSHA1", + MACSHA1: "anyMACSHA1", + MACMD5: "anyMACMD5", + IFA: "anyIFA", + IP: "1.2.3.4", + IPv6: "2001:0db8:0000:0000:0000:ff00:0042:8329", Geo: &openrtb.Geo{ - Lat: 123.456, + Lat: 123.46, Lon: 678.89, Metro: "some metro", City: "some city", ZIP: "some zip", }, }, - ipv6: ScrubStrategyIPV6Lowest16, - geo: ScrubStrategyGeoNone, + id: ScrubStrategyDeviceIDNone, + ipv4: ScrubStrategyIPV4None, + ipv6: ScrubStrategyIPV6None, + geo: ScrubStrategyGeoReducedPrecision, }, { - description: "IPv6 None & Geo None", + description: "Isolated - Geo - Full", expected: &openrtb.Device{ - DIDMD5: "", - DIDSHA1: "", - DPIDMD5: "", - DPIDSHA1: "", - MACSHA1: "", - MACMD5: "", - IFA: "", - IP: "1.2.3.0", + DIDMD5: "anyDIDMD5", + DIDSHA1: "anyDIDSHA1", + DPIDMD5: "anyDPIDMD5", + DPIDSHA1: "anyDPIDSHA1", + MACSHA1: "anyMACSHA1", + MACMD5: "anyMACMD5", + IFA: "anyIFA", + IP: "1.2.3.4", IPv6: "2001:0db8:0000:0000:0000:ff00:0042:8329", - Geo: &openrtb.Geo{ - Lat: 123.456, - Lon: 678.89, - Metro: "some metro", - City: "some city", - ZIP: "some zip", - }, + Geo: &openrtb.Geo{}, }, + id: ScrubStrategyDeviceIDNone, + ipv4: ScrubStrategyIPV4None, ipv6: ScrubStrategyIPV6None, - geo: ScrubStrategyGeoNone, + geo: ScrubStrategyGeoFull, }, } for _, test := range testCases { - result := NewScrubber().ScrubDevice(device, test.ipv6, test.geo) + result := NewScrubber().ScrubDevice(device, test.id, test.ipv4, test.ipv6, test.geo) assert.Equal(t, test.expected, result, test.description) } } func TestScrubDeviceNil(t *testing.T) { - result := NewScrubber().ScrubDevice(nil, ScrubStrategyIPV6None, ScrubStrategyGeoNone) + result := NewScrubber().ScrubDevice(nil, ScrubStrategyDeviceIDNone, ScrubStrategyIPV4None, ScrubStrategyIPV6None, ScrubStrategyGeoNone) assert.Nil(t, result) } @@ -458,7 +418,7 @@ func TestScrubIPV4(t *testing.T) { } for _, test := range testCases { - result := scrubIPV4(test.IP) + result := scrubIPV4Lowest8(test.IP) assert.Equal(t, test.cleanedIP, result, test.description) } } diff --git a/privacy/writer.go b/privacy/writer.go new file mode 100644 index 00000000000..a68a158ced8 --- /dev/null +++ b/privacy/writer.go @@ -0,0 +1,18 @@ +package privacy + +import ( + "github.com/PubMatic-OpenWrap/openrtb" +) + +// PolicyWriter mutates an OpenRTB bid request with a policy's regulatory information. +type PolicyWriter interface { + Write(req *openrtb.BidRequest) error +} + +// NilPolicyWriter implements the PolicyWriter interface but performs no action. +type NilPolicyWriter struct{} + +// Write is hardcoded to perform no action with the OpenRTB bid request. +func (NilPolicyWriter) Write(req *openrtb.BidRequest) error { + return nil +} diff --git a/privacy/writer_test.go b/privacy/writer_test.go new file mode 100644 index 00000000000..754e6ffe2c9 --- /dev/null +++ b/privacy/writer_test.go @@ -0,0 +1,25 @@ +package privacy + +import ( + "encoding/json" + "testing" + + "github.com/PubMatic-OpenWrap/openrtb" + "github.com/stretchr/testify/assert" +) + +func TestNilWriter(t *testing.T) { + request := &openrtb.BidRequest{ + ID: "anyID", + Ext: json.RawMessage(`{"anyJson":"anyValue"}`), + } + expectedRequest := &openrtb.BidRequest{ + ID: "anyID", + Ext: json.RawMessage(`{"anyJson":"anyValue"}`), + } + + nilWriter := &NilPolicyWriter{} + nilWriter.Write(request) + + assert.Equal(t, expectedRequest, request) +} diff --git a/router/admin.go b/router/admin.go index 608c7869e99..05281d56e15 100644 --- a/router/admin.go +++ b/router/admin.go @@ -3,12 +3,13 @@ package router import ( "net/http" "net/http/pprof" + "time" "github.com/PubMatic-OpenWrap/prebid-server/currencies" "github.com/PubMatic-OpenWrap/prebid-server/endpoints" ) -func Admin(revision string, rateConverter *currencies.RateConverter) *http.ServeMux { +func Admin(revision string, rateConverter *currencies.RateConverter, rateConverterFetchingInterval time.Duration) *http.ServeMux { // Add endpoints to the admin server // Making sure to add pprof routes mux := http.NewServeMux() @@ -19,7 +20,7 @@ func Admin(revision string, rateConverter *currencies.RateConverter) *http.Serve mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol) mux.HandleFunc("/debug/pprof/trace", pprof.Trace) // Register prebid-server defined admin handlers - mux.HandleFunc("/currency/rates", endpoints.NewCurrencyRatesEndpoint(rateConverter)) + mux.HandleFunc("/currency/rates", endpoints.NewCurrencyRatesEndpoint(rateConverter, rateConverterFetchingInterval)) mux.HandleFunc("/version", endpoints.NewVersionEndpoint(revision)) return mux } diff --git a/router/router.go b/router/router.go index 755c18a452b..33ef6b89528 100644 --- a/router/router.go +++ b/router/router.go @@ -12,9 +12,12 @@ import ( "strings" "time" - "github.com/prometheus/client_golang/prometheus" + "github.com/PubMatic-OpenWrap/prebid-server/endpoints" + "github.com/PubMatic-OpenWrap/prebid-server/endpoints/openrtb2" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/usersync" + "github.com/prometheus/client_golang/prometheus" "github.com/PubMatic-OpenWrap/prebid-server/adapters" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adform" @@ -34,8 +37,6 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/cache/postgrescache" "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/currencies" - "github.com/PubMatic-OpenWrap/prebid-server/endpoints" - "github.com/PubMatic-OpenWrap/prebid-server/endpoints/openrtb2" "github.com/PubMatic-OpenWrap/prebid-server/exchange" "github.com/PubMatic-OpenWrap/prebid-server/gdpr" "github.com/PubMatic-OpenWrap/prebid-server/openrtb_ext" @@ -44,7 +45,6 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/ssl" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" storedRequestsConf "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/config" - "github.com/PubMatic-OpenWrap/prebid-server/usersync" "github.com/PubMatic-OpenWrap/prebid-server/usersync/usersyncers" "github.com/golang/glog" @@ -59,6 +59,7 @@ var ( g_syncers map[openrtb_ext.BidderName]usersync.Usersyncer g_cfg *config.Configuration g_ex exchange.Exchange + g_accounts stored_requests.AccountFetcher g_paramsValidator openrtb_ext.BidderParamValidator g_storedReqFetcher stored_requests.Fetcher g_gdprPerms gdpr.Permissions @@ -207,6 +208,7 @@ func New(cfg *config.Configuration, rateConvertor *currencies.RateConverter) (r generalHttpClient := &http.Client{ Transport: &http.Transport{ + MaxConnsPerHost: cfg.Client.MaxConnsPerHost, MaxIdleConns: cfg.Client.MaxIdleConns, MaxIdleConnsPerHost: cfg.Client.MaxIdleConnsPerHost, IdleConnTimeout: time.Duration(cfg.Client.IdleConnTimeout) * time.Second, @@ -216,6 +218,7 @@ func New(cfg *config.Configuration, rateConvertor *currencies.RateConverter) (r cacheHttpClient := &http.Client{ Transport: &http.Transport{ + MaxConnsPerHost: cfg.CacheClient.MaxConnsPerHost, MaxIdleConns: cfg.CacheClient.MaxIdleConns, MaxIdleConnsPerHost: cfg.CacheClient.MaxIdleConnsPerHost, IdleConnTimeout: time.Duration(cfg.CacheClient.IdleConnTimeout) * time.Second, @@ -230,7 +233,7 @@ func New(cfg *config.Configuration, rateConvertor *currencies.RateConverter) (r var db *sql.DB // Metrics engine g_metrics = metricsConf.NewMetricsEngine(cfg, legacyBidderList) - db, _, g_storedReqFetcher, _, g_categoriesFetcher, g_videoFetcher = storedRequestsConf.NewStoredRequests(cfg, g_metrics, generalHttpClient, r.Router) + db, _, g_storedReqFetcher, _, g_accounts, g_categoriesFetcher, g_videoFetcher = storedRequestsConf.NewStoredRequests(cfg, g_metrics, generalHttpClient, r.Router) // todo(zachbadgett): better shutdown //r.Shutdown = shutdown @@ -260,22 +263,22 @@ func New(cfg *config.Configuration, rateConvertor *currencies.RateConverter) (r exchanges = newExchangeMap(cfg) g_cacheClient = pbc.NewClient(cacheHttpClient, &cfg.CacheURL, &cfg.ExtCacheURL, g_metrics) - g_ex = exchange.NewExchange(generalHttpClient, g_cacheClient, cfg, g_metrics, bidderInfos, g_gdprPerms, rateConvertor) + g_ex = exchange.NewExchange(generalHttpClient, g_cacheClient, cfg, g_metrics, bidderInfos, g_gdprPerms, rateConvertor, g_categoriesFetcher) /* - openrtbEndpoint, err := openrtb2.NewEndpoint(theExchange, paramsValidator, fetcher, categoriesFetcher, cfg, r.MetricsEngine, pbsAnalytics, disabledBidders, defReqJSON, activeBiddersMap) + openrtbEndpoint, err := openrtb2.NewEndpoint(theExchange, paramsValidator, fetcher, accounts, cfg, r.MetricsEngine, pbsAnalytics, disabledBidders, defReqJSON, activeBiddersMap) if err != nil { glog.Fatalf("Failed to create the openrtb endpoint handler. %v", err) } - ampEndpoint, err := openrtb2.NewAmpEndpoint(theExchange, paramsValidator, ampFetcher, categoriesFetcher, cfg, r.MetricsEngine, pbsAnalytics, disabledBidders, defReqJSON, activeBiddersMap) + ampEndpoint, err := openrtb2.NewAmpEndpoint(theExchange, paramsValidator, ampFetcher, accounts, cfg, r.MetricsEngine, pbsAnalytics, disabledBidders, defReqJSON, activeBiddersMap) if err != nil { glog.Fatalf("Failed to create the amp endpoint handler. %v", err) } - videoEndpoint, err := openrtb2.NewVideoEndpoint(theExchange, paramsValidator, fetcher, videoFetcher, categoriesFetcher, cfg, r.MetricsEngine, pbsAnalytics, disabledBidders, defReqJSON, activeBiddersMap, cacheClient) + videoEndpoint, err := openrtb2.NewVideoEndpoint(theExchange, paramsValidator, fetcher, videoFetcher, accounts, cfg, r.MetricsEngine, pbsAnalytics, disabledBidders, defReqJSON, activeBiddersMap, cacheClient) if err != nil { glog.Fatalf("Failed to create the video endpoint handler. %v", err) } @@ -297,6 +300,16 @@ func New(cfg *config.Configuration, rateConvertor *currencies.RateConverter) (r r.GET("/", serveIndex) r.ServeFiles("/static/*filepath", http.Dir("static")) + // vtrack endpoint + if cfg.VTrack.Enabled { + vtrackEndpoint := events.NewVTrackEndpoint(cfg, accounts, cacheClient, bidderInfos) + r.POST("/vtrack", vtrackEndpoint) + } + + // event endpoint + eventEndpoint := events.NewEventEndpoint(cfg, accounts, pbsAnalytics) + r.GET("/event", eventEndpoint) + userSyncDeps := &pbs.UserSyncDeps{ HostCookieConfig: &(cfg.HostCookie), ExternalUrl: cfg.ExternalURL, @@ -309,13 +322,14 @@ func New(cfg *config.Configuration, rateConvertor *currencies.RateConverter) (r r.GET("/getuids", endpoints.NewGetUIDsEndpoint(cfg.HostCookie)) r.POST("/optout", userSyncDeps.OptOut) r.GET("/optout", userSyncDeps.OptOut) + */ return r, nil } //OrtbAuctionEndpointWrapper Openwrap wrapper method for calling /openrtb2/auction endpoint func OrtbAuctionEndpointWrapper(w http.ResponseWriter, r *http.Request) error { - ortbAuctionEndpoint, err := openrtb2.NewEndpoint(g_ex, g_paramsValidator, g_storedReqFetcher, g_categoriesFetcher, g_cfg, g_metrics, g_analytics, g_disabledBidders, g_defReqJSON, g_bidderMap) + ortbAuctionEndpoint, err := openrtb2.NewEndpoint(g_ex, g_paramsValidator, g_storedReqFetcher, g_accounts, g_cfg, g_metrics, g_analytics, g_disabledBidders, g_defReqJSON, g_bidderMap) if err != nil { return err } @@ -325,7 +339,7 @@ func OrtbAuctionEndpointWrapper(w http.ResponseWriter, r *http.Request) error { //VideoAuctionEndpointWrapper Openwrap wrapper method for calling /openrtb2/video endpoint func VideoAuctionEndpointWrapper(w http.ResponseWriter, r *http.Request) error { - videoAuctionEndpoint, err := openrtb2.NewCTVEndpoint(g_ex, g_paramsValidator, g_storedReqFetcher, g_videoFetcher, g_categoriesFetcher, g_cfg, g_metrics, g_analytics, g_disabledBidders, g_defReqJSON, g_bidderMap) + videoAuctionEndpoint, err := openrtb2.NewCTVEndpoint(g_ex, g_paramsValidator, g_storedReqFetcher, g_videoFetcher, g_accounts, g_cfg, g_metrics, g_analytics, g_disabledBidders, g_defReqJSON, g_bidderMap) if err != nil { return err } diff --git a/static/bidder-info/33across.yaml b/static/bidder-info/33across.yaml index 84ba6d68611..67e6996accf 100644 --- a/static/bidder-info/33across.yaml +++ b/static/bidder-info/33across.yaml @@ -4,6 +4,8 @@ capabilities: app: mediaTypes: - banner + - video site: mediaTypes: - banner + - video diff --git a/static/bidder-info/acuityads.yaml b/static/bidder-info/acuityads.yaml new file mode 100644 index 00000000000..9da1446d918 --- /dev/null +++ b/static/bidder-info/acuityads.yaml @@ -0,0 +1,14 @@ +maintainer: + email: "integrations@acuityads.com" +capabilities: + app: + mediaTypes: + - banner + - video + - native + site: + mediaTypes: + - banner + - video + - native + diff --git a/static/bidder-info/adform.yaml b/static/bidder-info/adform.yaml index 8aafd9f6815..4dce10b9af8 100644 --- a/static/bidder-info/adform.yaml +++ b/static/bidder-info/adform.yaml @@ -4,6 +4,8 @@ capabilities: app: mediaTypes: - banner + - video site: mediaTypes: - banner + - video diff --git a/static/bidder-info/adman.yaml b/static/bidder-info/adman.yaml new file mode 100644 index 00000000000..932ef2e4242 --- /dev/null +++ b/static/bidder-info/adman.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "prebid@admanmedia.com" +capabilities: + app: + mediaTypes: + - banner + - video + site: + mediaTypes: + - banner + - video \ No newline at end of file diff --git a/static/bidder-info/adprime.yaml b/static/bidder-info/adprime.yaml new file mode 100644 index 00000000000..9759ed63be7 --- /dev/null +++ b/static/bidder-info/adprime.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "rafal@adprime.com" +capabilities: + app: + mediaTypes: + - banner + - video + site: + mediaTypes: + - banner + - video \ No newline at end of file diff --git a/static/bidder-info/adtelligent.yaml b/static/bidder-info/adtelligent.yaml index fe791343daf..7a20d52b266 100644 --- a/static/bidder-info/adtelligent.yaml +++ b/static/bidder-info/adtelligent.yaml @@ -4,6 +4,7 @@ capabilities: app: mediaTypes: - banner + - video site: mediaTypes: - banner diff --git a/static/bidder-info/amx.yaml b/static/bidder-info/amx.yaml new file mode 100644 index 00000000000..3e20d2095f6 --- /dev/null +++ b/static/bidder-info/amx.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "prebid@amxrtb.com" +capabilities: + site: + mediaTypes: + - banner + - video + app: + mediaTypes: + - banner + - video \ No newline at end of file diff --git a/static/bidder-info/audienceNetwork.yaml b/static/bidder-info/audienceNetwork.yaml index 56230bf3f9a..324e5c6dff8 100644 --- a/static/bidder-info/audienceNetwork.yaml +++ b/static/bidder-info/audienceNetwork.yaml @@ -1,11 +1,6 @@ maintainer: email: "none" capabilities: - site: - mediaTypes: - - banner - - video - - native app: mediaTypes: - banner diff --git a/static/bidder-info/between.yaml b/static/bidder-info/between.yaml new file mode 100644 index 00000000000..d317d275c59 --- /dev/null +++ b/static/bidder-info/between.yaml @@ -0,0 +1,9 @@ +maintainer: + email: "buying@betweenx.com" +capabilities: + site: + mediaTypes: + - banner + app: + mediaTypes: + - banner \ No newline at end of file diff --git a/static/bidder-info/colossus.yaml b/static/bidder-info/colossus.yaml new file mode 100644 index 00000000000..901c824c603 --- /dev/null +++ b/static/bidder-info/colossus.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "support@huddledmasses.com" +capabilities: + app: + mediaTypes: + - banner + - video + site: + mediaTypes: + - banner + - video diff --git a/static/bidder-info/connectad.yaml b/static/bidder-info/connectad.yaml new file mode 100644 index 00000000000..1b3e593d78d --- /dev/null +++ b/static/bidder-info/connectad.yaml @@ -0,0 +1,9 @@ +maintainer: + email: "support@connectad.io" +capabilities: + app: + mediaTypes: + - banner + site: + mediaTypes: + - banner diff --git a/static/bidder-info/emx_digital.yaml b/static/bidder-info/emx_digital.yaml index 40f73fd000f..49a068093eb 100644 --- a/static/bidder-info/emx_digital.yaml +++ b/static/bidder-info/emx_digital.yaml @@ -4,3 +4,8 @@ capabilities: site: mediaTypes: - banner + - video + app: + mediaTypes: + - banner + - video \ No newline at end of file diff --git a/static/bidder-info/grid.yaml b/static/bidder-info/grid.yaml index 9594830c0d0..325421a2c05 100644 --- a/static/bidder-info/grid.yaml +++ b/static/bidder-info/grid.yaml @@ -1,7 +1,11 @@ maintainer: email: "grid-tech@themediagrid.com" capabilities: - site: + app: mediaTypes: - banner - video + site: + mediaTypes: + - banner + - video \ No newline at end of file diff --git a/static/bidder-info/gumgum.yaml b/static/bidder-info/gumgum.yaml index 0feca7cdf73..6ba563b4c1c 100644 --- a/static/bidder-info/gumgum.yaml +++ b/static/bidder-info/gumgum.yaml @@ -4,3 +4,4 @@ capabilities: site: mediaTypes: - banner + - video \ No newline at end of file diff --git a/static/bidder-info/inmobi.yaml b/static/bidder-info/inmobi.yaml new file mode 100644 index 00000000000..3f8cdd8cb91 --- /dev/null +++ b/static/bidder-info/inmobi.yaml @@ -0,0 +1,8 @@ +maintainer: + email: "prebid-support@inmobi.com" + +capabilities: + app: + mediaTypes: + - banner + - video diff --git a/static/bidder-info/invibes.yaml b/static/bidder-info/invibes.yaml new file mode 100644 index 00000000000..1432529787e --- /dev/null +++ b/static/bidder-info/invibes.yaml @@ -0,0 +1,6 @@ +maintainer: + email: "system_operations@invibes.com" +capabilities: + site: + mediaTypes: + - banner diff --git a/static/bidder-info/krushmedia.yaml b/static/bidder-info/krushmedia.yaml new file mode 100644 index 00000000000..342e11df2c7 --- /dev/null +++ b/static/bidder-info/krushmedia.yaml @@ -0,0 +1,13 @@ +maintainer: + email: "adapter@krushmedia.com" +capabilities: + app: + mediaTypes: + - banner + - video + - native + site: + mediaTypes: + - banner + - video + - native \ No newline at end of file diff --git a/static/bidder-info/logicad.yaml b/static/bidder-info/logicad.yaml new file mode 100644 index 00000000000..c087516c061 --- /dev/null +++ b/static/bidder-info/logicad.yaml @@ -0,0 +1,10 @@ +maintainer: + email: "prebid@so-netmedia.jp" +capabilities: + site: + mediaTypes: + - banner + app: + mediaTypes: + - banner + diff --git a/static/bidder-info/nobid.yaml b/static/bidder-info/nobid.yaml new file mode 100644 index 00000000000..51a55de46bc --- /dev/null +++ b/static/bidder-info/nobid.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "developers@nobid.io" +capabilities: + site: + mediaTypes: + - banner + - video + app: + mediaTypes: + - banner + - video diff --git a/static/bidder-info/silvermob.yaml b/static/bidder-info/silvermob.yaml new file mode 100644 index 00000000000..5f1e4809dd3 --- /dev/null +++ b/static/bidder-info/silvermob.yaml @@ -0,0 +1,8 @@ +maintainer: + email: "support@silvermob.com" +capabilities: + app: + mediaTypes: + - banner + - video + - native \ No newline at end of file diff --git a/static/bidder-info/smaato.yaml b/static/bidder-info/smaato.yaml new file mode 100644 index 00000000000..db3e61e5cc6 --- /dev/null +++ b/static/bidder-info/smaato.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "prebid@smaato.com" +capabilities: + app: + mediaTypes: + - banner + - video + site: + mediaTypes: + - banner + - video diff --git a/static/bidder-info/smartadserver.yaml b/static/bidder-info/smartadserver.yaml new file mode 100644 index 00000000000..626b7dac00d --- /dev/null +++ b/static/bidder-info/smartadserver.yaml @@ -0,0 +1,11 @@ +maintainer: + email: "support@smartadserver.com" +capabilities: + app: + mediaTypes: + - banner + - video + site: + mediaTypes: + - banner + - video diff --git a/static/bidder-info/smartyads.yaml b/static/bidder-info/smartyads.yaml new file mode 100644 index 00000000000..df4c1b7ffb5 --- /dev/null +++ b/static/bidder-info/smartyads.yaml @@ -0,0 +1,14 @@ +maintainer: + email: "support@smartyads.com" +capabilities: + app: + mediaTypes: + - banner + - video + - native + site: + mediaTypes: + - banner + - video + - native + diff --git a/static/bidder-info/yieldmo.yaml b/static/bidder-info/yieldmo.yaml index 514f17455ea..64cda519acd 100644 --- a/static/bidder-info/yieldmo.yaml +++ b/static/bidder-info/yieldmo.yaml @@ -1,6 +1,11 @@ maintainer: email: "prebid@yieldmo.com" capabilities: + app: + mediaTypes: + - banner + - video site: mediaTypes: - banner + - video diff --git a/static/bidder-params/acuityads.json b/static/bidder-params/acuityads.json new file mode 100644 index 00000000000..bae86ad2103 --- /dev/null +++ b/static/bidder-params/acuityads.json @@ -0,0 +1,19 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "AcuityAds Adapter Params", + "description": "A schema which validates params accepted by the AcuityAds adapter", + "type": "object", + "properties": { + "host": { + "type": "string", + "description": "Network host to send request", + "minLength": 1 + }, + "accountid": { + "type": "string", + "description": "Account id", + "minLength": 1 + } + }, + "required": ["host", "accountid"] +} \ No newline at end of file diff --git a/static/bidder-params/adform.json b/static/bidder-params/adform.json index 67f09623ee4..f0b8c7a6be0 100644 --- a/static/bidder-params/adform.json +++ b/static/bidder-params/adform.json @@ -22,6 +22,20 @@ "type": "string", "description": "Comma-separated keywords. Forbidden symbols: &.", "pattern": "^[^&]*$" + }, + "cdims": { + "type": "string", + "description": "Comma-separated creative dimentions.", + "pattern": "(^\\d+x\\d+)(,\\d+x\\d+)*$" + }, + "minp": { + "type": "number", + "description": "The minimum CPM price.", + "minimum": 0 + }, + "url": { + "type": "string", + "description": "Custom URL for targeting." } }, "required": ["mid"] diff --git a/static/bidder-params/adman.json b/static/bidder-params/adman.json new file mode 100644 index 00000000000..90021e2cdfd --- /dev/null +++ b/static/bidder-params/adman.json @@ -0,0 +1,15 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Adman Adapter Params", + "description": "A schema which validates params accepted by the Adman adapter", + + "type": "object", + "properties": { + "TagID": { + "type": "string", + "description": "An ID which identifies the adman ad tag" + } + }, + "required" : [ "TagID" ] + } + \ No newline at end of file diff --git a/static/bidder-params/adoppler.json b/static/bidder-params/adoppler.json index c2bdde4f60f..508eef478c0 100644 --- a/static/bidder-params/adoppler.json +++ b/static/bidder-params/adoppler.json @@ -7,6 +7,10 @@ "adunit": { "type": "string", "description": "AdUnit to bid against to." + }, + "client": { + "type": "string", + "description": "Client name." } }, "required": ["adunit"] diff --git a/static/bidder-params/adprime.json b/static/bidder-params/adprime.json new file mode 100644 index 00000000000..d527056597d --- /dev/null +++ b/static/bidder-params/adprime.json @@ -0,0 +1,14 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Adprime Adapter Params", + "description": "A schema which validates params accepted by the Adprime adapter", + + "type": "object", + "properties": { + "TagID": { + "type": "string", + "description": "An ID which identifies the adprime ad tag" + } + }, + "required" : [ "TagID" ] + } \ No newline at end of file diff --git a/static/bidder-params/amx.json b/static/bidder-params/amx.json new file mode 100644 index 00000000000..f9b1b26b3db --- /dev/null +++ b/static/bidder-params/amx.json @@ -0,0 +1,16 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "AMX RTB Adapter Params", + "description": "A schema to validate params accepted by the AMX adapter", + "type": "object", + "properties": { + "tagId" : { + "type": "string", + "description": "Set a tagId (overrides site.publisher.id, or app.publisher.id)" + }, + "adUnitId": { + "type": "string", + "description": "Override imp.tagid value to provide a custom value in AMX ad unit ID reporting" + } + } +} diff --git a/static/bidder-params/between.json b/static/bidder-params/between.json new file mode 100644 index 00000000000..88bbf087be9 --- /dev/null +++ b/static/bidder-params/between.json @@ -0,0 +1,14 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "BetweenDigital Adapter Params", + "description": "A schema which validates params accepted by the BetweenDigital adapter", + "type": "object", + "properties": { + "host": { + "type": "string", + "description": "Network Host to request from" + } + + }, + "required": ["host"] +} diff --git a/static/bidder-params/colossus.json b/static/bidder-params/colossus.json new file mode 100644 index 00000000000..f2732fa0854 --- /dev/null +++ b/static/bidder-params/colossus.json @@ -0,0 +1,14 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Colossus Adapter Params", + "description": "A schema which validates params accepted by the Colossus adapter", + + "type": "object", + "properties": { + "TagID": { + "type": "string", + "description": "An ID which identifies the colossus ad tag" + } + }, + "required" : [ "TagID" ] + } diff --git a/static/bidder-params/connectad.json b/static/bidder-params/connectad.json new file mode 100644 index 00000000000..961b3b71202 --- /dev/null +++ b/static/bidder-params/connectad.json @@ -0,0 +1,24 @@ + +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "ConnectAd S2S dapter Params", + "description": "A schema which validates params accepted by the ConnectAd Adapter", + + "type": "object", + "properties": { + "networkId": { + "type": "integer", + "description": "NetworkId" + }, + "siteId": { + "type": "integer", + "description": "SiteId" + }, + "bidfloor": { + "type": "number", + "description": "Requestes Floorprice" + } + }, + "required": ["networkId", "siteId"] + } + \ No newline at end of file diff --git a/static/bidder-params/conversant.json b/static/bidder-params/conversant.json index 4f7200acd3b..ba9c6bd584d 100644 --- a/static/bidder-params/conversant.json +++ b/static/bidder-params/conversant.json @@ -24,10 +24,6 @@ "type": "integer", "description": "Ad position on screen." }, - "mobile": { - "type": "integer", - "description": "Indicate if the site is mobile optimized." - }, "mimes": { "type": "array", "description": "Array of content MIME types. For videos only.", diff --git a/static/bidder-params/inmobi.json b/static/bidder-params/inmobi.json new file mode 100644 index 00000000000..631b3137b72 --- /dev/null +++ b/static/bidder-params/inmobi.json @@ -0,0 +1,13 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "InMobi Adapter Params", + "description": "A schema which validates params accepted by the InMobi adapter", + "type": "object", + "properties": { + "plc": { + "type": ["string"], + "description": "An ID corresponding to the placement selling the impression" + } + }, + "required": ["plc"] +} diff --git a/static/bidder-params/invibes.json b/static/bidder-params/invibes.json new file mode 100644 index 00000000000..11d276f8d3e --- /dev/null +++ b/static/bidder-params/invibes.json @@ -0,0 +1,30 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Invibes Adapter Params", + "description": "A schema which validates params accepted by the Invibes adapter", + "type": "object", + "properties": { + "placementId": { + "type": "string", + "minLength": 1, + "description": "An ID which identifies the site selling the impression" + }, + "domainId": { + "type": "integer", + "description": "Ad domain id" + }, + "debug": { + "type": "object", + "properties": { + "testBvid": { + "type": "string" + }, + "testLog": { + "type": "boolean" + } + }, + "description": "Parameters used for debugging porposes" + } + }, + "required": ["placementId"] +} diff --git a/static/bidder-params/krushmedia.json b/static/bidder-params/krushmedia.json new file mode 100644 index 00000000000..e395da85617 --- /dev/null +++ b/static/bidder-params/krushmedia.json @@ -0,0 +1,13 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Krushmedia Adapter Params", + "description": "A schema which validates params accepted by the Krushmedia adapter", + "type": "object", + "properties": { + "key": { + "type": "string", + "description": "ssp key" + } + }, + "required": ["key"] + } \ No newline at end of file diff --git a/static/bidder-params/kubient.json b/static/bidder-params/kubient.json index a75dd734ff2..9b975289a7b 100644 --- a/static/bidder-params/kubient.json +++ b/static/bidder-params/kubient.json @@ -3,5 +3,11 @@ "title": "Kubient Adapter Params", "description": "A schema which validates params accepted by the Kubient adapter", "type": "object", - "properties": { } + "properties": { + "zoneid": { + "type": "string", + "description": "Zone ID identifies Kubient placement ID.", + "minLength": 1 + } + } } diff --git a/static/bidder-params/logicad.json b/static/bidder-params/logicad.json new file mode 100644 index 00000000000..2a892f91266 --- /dev/null +++ b/static/bidder-params/logicad.json @@ -0,0 +1,13 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Logicad Adapter Params", + "description": "A schema which validates params accepted by the Logicad adapter", + "type": "object", + "properties": { + "tid": { + "type": "string", + "description": "Logicad for Publishers placement ID" + } + }, + "required": ["tid"] +} diff --git a/static/bidder-params/nobid.json b/static/bidder-params/nobid.json new file mode 100644 index 00000000000..576dbfecb5c --- /dev/null +++ b/static/bidder-params/nobid.json @@ -0,0 +1,18 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "NoBid Adapter Params", + "description": "A schema which validates params accepted by the NoBid adapter", + + "type": "object", + "properties": { + "siteId": { + "type": "integer", + "description": "A Required ID which identifies the NoBid site. The siteId paramerter is provided by your NoBid account manager." + }, "placementId": { + "type": "integer", + "description": "An oprional ID which identifies an adunit in a site. The placementId paramerter is provided by your NoBid account manager." + } + }, + "required": ["siteId"] + } + \ No newline at end of file diff --git a/static/bidder-params/openx.json b/static/bidder-params/openx.json index 93a672ed629..6dbd10178e4 100644 --- a/static/bidder-params/openx.json +++ b/static/bidder-params/openx.json @@ -16,6 +16,11 @@ "pattern": "\\.[a-zA-Z]{2,3}$", "format": "hostname" }, + "platform": { + "type": "string", + "description": "The platform id for the customer.", + "format": "uuid" + }, "customFloor": { "type": "number", "description": "The minimum CPM price in USD.", @@ -26,6 +31,19 @@ "description": "User-defined targeting key-value pairs." } }, - - "required": ["unit", "delDomain"] + "required": [ + "unit" + ], + "anyOf": [ + { + "required": [ + "delDomain" + ] + }, + { + "required": [ + "platform" + ] + } + ] } diff --git a/static/bidder-params/silvermob.json b/static/bidder-params/silvermob.json new file mode 100644 index 00000000000..8ebc85a2ab7 --- /dev/null +++ b/static/bidder-params/silvermob.json @@ -0,0 +1,17 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "SilverMob Adapter Params", + "description": "A schema which validates params accepted by the SilverMob adapter", + "type": "object", + "properties": { + "zoneid": { + "type": "string", + "description": "Zone ID" + }, + "host": { + "type": "string", + "description": "Host" + } + }, + "required": ["zoneid", "host"] + } \ No newline at end of file diff --git a/static/bidder-params/smaato.json b/static/bidder-params/smaato.json new file mode 100644 index 00000000000..aa91c4bacc5 --- /dev/null +++ b/static/bidder-params/smaato.json @@ -0,0 +1,17 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Smaato Adapter Params", + "description": "A schema which validates params accepted by the Smaato adapter", + "type": "object", + "properties": { + "publisherId": { + "type": "string", + "description": "A unique identifier for this impression within the context of the bid request" + }, + "adspaceId": { + "type": "string", + "description": "Identifier for specific ad placement is SOMA `adspaceId`" + } + }, + "required": ["publisherId","adspaceId"] +} \ No newline at end of file diff --git a/static/bidder-params/smartadserver.json b/static/bidder-params/smartadserver.json new file mode 100644 index 00000000000..b76a3bd6ac9 --- /dev/null +++ b/static/bidder-params/smartadserver.json @@ -0,0 +1,35 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Smartadserver Adapter Params", + "description": "A schema which validates params accepted by the Smartadserver adapter", + + "type": "object", + "properties": { + "siteId": { + "type": "integer", + "description": "The site id.", + "minimum": 1 + }, + "pageId": { + "type": "integer", + "description": "The page id.", + "minimum": 1 + }, + "formatId": { + "type": "integer", + "description": "The format id.", + "minimum": 1 + }, + "networkId": { + "type": "integer", + "description": "The network id.", + "minimum": 1 + } + }, + "dependencies": { + "siteId": { "required": ["pageId", "formatId"] }, + "pageId": { "required": ["siteId", "formatId"] }, + "formatId": { "required": ["siteId", "pageId"] } + }, + "required": ["networkId"] +} \ No newline at end of file diff --git a/static/bidder-params/smartyads.json b/static/bidder-params/smartyads.json new file mode 100644 index 00000000000..629fdde3263 --- /dev/null +++ b/static/bidder-params/smartyads.json @@ -0,0 +1,21 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "SmartyAds Adapter Params", + "description": "A schema which validates params accepted by the SmartyAds adapter", + "type": "object", + "properties": { + "host": { + "type": "string", + "description": "Network host to send request" + }, + "sourceid": { + "type": "string", + "description": "Partner id" + }, + "accountid": { + "type": "string", + "description": "Account id" + } + }, + "required": ["host", "sourceid", "accountid"] +} \ No newline at end of file diff --git a/static/category-mapping/freewheel/freewheel.json b/static/category-mapping/freewheel/freewheel.json index 1c4a4fa2471..1b849b5392d 100644 --- a/static/category-mapping/freewheel/freewheel.json +++ b/static/category-mapping/freewheel/freewheel.json @@ -1178,5 +1178,81 @@ "IAB22-3": { "id": "410", "name": "Product" + }, + "IAB1": { + "id": "392", + "name": "Entertainment" + }, + "IAB2": { + "id": "399", + "name": "Automotive" + }, + "IAB3": { + "id": "393", + "name": "Business Services" + }, + "IAB4": { + "id": "405", + "name": "Educational Services" + }, + "IAB5": { + "id": "405", + "name": "Educational Services" + }, + "IAB7": { + "id": "406", + "name": "Health Care Services" + }, + "IAB8": { + "id": "394", + "name": "Food" + }, + "IAB9": { + "id": "392", + "name": "Entertainment" + }, + "IAB10": { + "id": "434", + "name": "Home Furnishings" + }, + "IAB11": { + "id": "398", + "name": "Government/Municipal" + }, + "IAB12": { + "id": "438", + "name": "News" + }, + "IAB13": { + "id": "393", + "name": "Business Services" + }, + "IAB16": { + "id": "423", + "name": "Pet Food/Supplies" + }, + "IAB17": { + "id": "425", + "name": "Professional Sports" + }, + "IAB18": { + "id": "397", + "name": "Apparel" + }, + "IAB19": { + "id": "409", + "name": "Computing Product" + }, + "IAB20": { + "id": "395", + "name": "Travel/Hotel/Airlines" + }, + "IAB21": { + "id": "416", + "name": "Real Estate" + }, + "IAB22": { + "id": "403", + "name": "Retail Stores/Chains" } -} \ No newline at end of file +} diff --git a/static/tcf1/fallback_gvl.json b/static/tcf1/fallback_gvl.json new file mode 100644 index 00000000000..9f1c8506b32 --- /dev/null +++ b/static/tcf1/fallback_gvl.json @@ -0,0 +1 @@ +{"vendorListVersion":215,"lastUpdated":"2020-08-13T16:00:19Z","purposes":[{"id":1,"name":"Information storage and access","description":"The storage of information, or access to information that is already stored, on your device such as advertising identifiers, device identifiers, cookies, and similar technologies."},{"id":2,"name":"Personalisation","description":"The collection and processing of information about your use of this service to subsequently personalise advertising and/or content for you in other contexts, such as on other websites or apps, over time. Typically, the content of the site or app is used to make inferences about your interests, which inform future selection of advertising and/or content."},{"id":3,"name":"Ad selection, delivery, reporting","description":"The collection of information, and combination with previously collected information, to select and deliver advertisements for you, and to measure the delivery and effectiveness of such advertisements. This includes using previously collected information about your interests to select ads, processing data about what advertisements were shown, how often they were shown, when and where they were shown, and whether you took any action related to the advertisement, including for example clicking an ad or making a purchase. This does not include personalisation, which is the collection and processing of information about your use of this service to subsequently personalise advertising and/or content for you in other contexts, such as websites or apps, over time."},{"id":4,"name":"Content selection, delivery, reporting","description":"The collection of information, and combination with previously collected information, to select and deliver content for you, and to measure the delivery and effectiveness of such content. This includes using previously collected information about your interests to select content, processing data about what content was shown, how often or how long it was shown, when and where it was shown, and whether the you took any action related to the content, including for example clicking on content. This does not include personalisation, which is the collection and processing of information about your use of this service to subsequently personalise content and/or advertising for you in other contexts, such as websites or apps, over time."},{"id":5,"name":"Measurement","description":"The collection of information about your use of the content, and combination with previously collected information, used to measure, understand, and report on your usage of the service. This does not include personalisation, the collection of information about your use of this service to subsequently personalise content and/or advertising for you in other contexts, i.e. on other service, such as websites or apps, over time."}],"features":[{"id":1,"name":"Matching Data to Offline Sources","description":"Combining data from offline sources that were initially collected in other contexts."},{"id":2,"name":"Linking Devices","description":"Allow processing of a user's data to connect such user across multiple devices."},{"id":3,"name":"Precise Geographic Location Data","description":"Allow processing of a user's precise geographic location data in support of a purpose for which that certain third party has consent."}],"vendors":[{"id":8,"name":"Emerse Sverige AB","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[1,2],"policyUrl":"https://www.emerse.com/privacy-policy/"},{"id":9,"name":"AdMaxim Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://www.admaxim.com/admaxim-privacy-policy/","deletedDate":"2020-06-17T00:00:00Z"},{"id":12,"name":"BeeswaxIO Corporation","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.beeswax.com/privacy/"},{"id":28,"name":"TripleLift, Inc.","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[3],"policyUrl":"https://triplelift.com/privacy/"},{"id":27,"name":"ADventori SAS","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.adventori.com/with-us/legal-notice/"},{"id":25,"name":"Verizon Media EMEA Limited","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[1,2,3],"policyUrl":"https://www.verizonmedia.com/policies/ie/en/verizonmedia/privacy/index.html"},{"id":26,"name":"Venatus Media Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.venatusmedia.com/privacy/"},{"id":1,"name":"Exponential Interactive, Inc d/b/a VDX.tv","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://vdx.tv/privacy/"},{"id":6,"name":"AdSpirit GmbH","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.adspirit.de/privacy"},{"id":30,"name":"BidTheatre AB","purposeIds":[1,2],"legIntPurposeIds":[3],"featureIds":[2,3],"policyUrl":"https://www.bidtheatre.com/privacy-policy"},{"id":24,"name":"Epsilon","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.conversantmedia.eu/legal/privacy-policy"},{"id":29,"name":"Etarget SE","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://www.etarget.sk/privacy.php","deletedDate":"2020-06-01T00:00:00Z"},{"id":39,"name":"ADITION technologies AG","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.adition.com/datenschutz"},{"id":11,"name":"Quantcast International Limited","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://www.quantcast.com/privacy/"},{"id":15,"name":"Adikteev","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.adikteev.com/privacy-policy-eng/"},{"id":4,"name":"Roq.ad Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.roq.ad/privacy-policy"},{"id":7,"name":"Vibrant Media Limited","purposeIds":[2,3,4,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://www.vibrantmedia.com/en/privacy-policy/"},{"id":2,"name":"Captify Technologies Limited","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"http://www.captify.co.uk/privacy-policy/"},{"id":37,"name":"NEURAL.ONE","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://web.neural.one/privacy-policy/"},{"id":13,"name":"Sovrn Holdings Inc","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.sovrn.com/sovrn-privacy/"},{"id":34,"name":"NEORY GmbH","purposeIds":[1,2,4,5],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://www.neory.com/privacy.html"},{"id":32,"name":"Xandr, Inc.","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[2,3],"policyUrl":"https://www.xandr.com/privacy/platform-privacy-policy/"},{"id":10,"name":"Index Exchange, Inc. ","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[2,3],"policyUrl":"https://www.indexexchange.com/privacy"},{"id":57,"name":"ADARA MEDIA UNLIMITED","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://adara.com/privacy-promise/"},{"id":63,"name":"Avocet Systems Limited","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2,3],"policyUrl":"https://avocet.io/privacy-portal"},{"id":51,"name":"xAd, Inc. dba GroundTruth","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.groundtruth.com/privacy-policy/"},{"id":49,"name":"TRADELAB","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://tradelab.com/en/privacy/"},{"id":45,"name":"Smart Adserver","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"https://smartadserver.com/end-user-privacy-policy/"},{"id":52,"name":"The Rubicon Project, Inc. ","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[3],"policyUrl":"http://www.rubiconproject.com/rubicon-project-yield-optimization-privacy-policy/"},{"id":71,"name":"Roku Advertising Services","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://docs.roku.com/published/userprivacypolicy/en/us"},{"id":79,"name":"MediaMath, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"http://www.mediamath.com/privacy-policy/"},{"id":91,"name":"Criteo SA","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.criteo.com/privacy/"},{"id":85,"name":"Crimtan Holdings Limited","purposeIds":[2],"legIntPurposeIds":[1,3,5],"featureIds":[1,3],"policyUrl":"https://crimtan.com/privacy/"},{"id":16,"name":"RTB House S.A.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.rtbhouse.com/privacy-center/services-privacy-policy/"},{"id":86,"name":"Scene Stealer Limited","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[3],"policyUrl":"http://scenestealer.tv/privacy-policy/"},{"id":94,"name":"Blis Media Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.blis.com/privacy/"},{"id":73,"name":"Simplifi Holdings Inc.","purposeIds":[2,3,4,5],"legIntPurposeIds":[1],"featureIds":[2,3],"policyUrl":"https://simpli.fi/site-privacy-policy/"},{"id":33,"name":"ShareThis, Inc","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"https://sharethis.com/privacy/"},{"id":20,"name":"N Technologies Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[2,3],"policyUrl":"https://n.rich/privacy-notice"},{"id":55,"name":"Madison Logic, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.madisonlogic.com/privacy/"},{"id":53,"name":"Sirdata","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.sirdata.com/privacy/"},{"id":69,"name":"OpenX","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.openx.com/legal/privacy-policy/"},{"id":98,"name":"GroupM UK Limited","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://www.groupm.com/privacy-notice"},{"id":62,"name":"Justpremium BV","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://justpremium.com/privacy-policy/"},{"id":19,"name":"Intent Media, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[2],"policyUrl":"https://intentmedia.com/privacy-policy/","deletedDate":"2020-07-17T00:00:00Z"},{"id":43,"name":"Vdopia DBA Chocolate Platform","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://chocolateplatform.com/privacy-policy/","deletedDate":"2019-08-06T00:00:00Z"},{"id":36,"name":"RhythmOne DBA Unruly Group Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.rhythmone.com/privacy-policy"},{"id":80,"name":"Sharethrough, Inc","purposeIds":[3,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://platform-cdn.sharethrough.com/privacy-policy"},{"id":81,"name":"PulsePoint, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.pulsepoint.com/privacy-policy/website","deletedDate":"2020-07-06T00:00:00Z"},{"id":23,"name":"Amobee, Inc. ","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.amobee.com/trust/privacy-guidelines"},{"id":35,"name":"Purch Group, Inc.","purposeIds":[1],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"http://www.purch.com/privacy-policy/","deletedDate":"2019-05-30T00:00:00Z"},{"id":3,"name":"affilinet","purposeIds":[2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.affili.net/de/footeritem/datenschutz","deletedDate":"2019-06-21T00:00:00Z"},{"id":74,"name":"Admotion SRL","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.admotion.com/policy/","deletedDate":"2019-07-24T00:00:00Z"},{"id":191,"name":"realzeit GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://realzeitmedia.com/privacy.html","deletedDate":"2019-04-29T00:00:00Z"},{"id":197,"name":"Switch Concepts Limited","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.switchconcepts.com/privacy-policy","deletedDate":"2019-07-26T00:00:00Z"},{"id":390,"name":"Parsec Media Inc.","purposeIds":[1,3],"legIntPurposeIds":[5],"featureIds":[1,3],"policyUrl":"www.parsec.media/privacy-policy","deletedDate":"2019-06-27T00:00:00Z"},{"id":459,"name":"uppr GmbH","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[],"policyUrl":"https://netzwerk.uppr.de/privacy-policy.do","deletedDate":"2019-06-17T00:00:00Z"},{"id":221,"name":"LEMO MEDIA GROUP LIMITED","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://www.lemomedia.com/terms.pdf","deletedDate":"2019-06-28T00:00:00Z"},{"id":478,"name":"RevLifter Ltd","purposeIds":[1],"legIntPurposeIds":[2],"featureIds":[],"policyUrl":"https://www.revlifter.com/privacy-policy","deletedDate":"2019-07-15T00:00:00Z"},{"id":500,"name":"Turbo","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"http://www.turboadv.com/white-rabbit-privacy-policy/","deletedDate":"2019-07-12T00:00:00Z"},{"id":68,"name":"Sizmek by Amazon","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[2],"policyUrl":"https://www.sizmek.com/privacy-policy/"},{"id":75,"name":"M32 Connect Inc","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://m32.media/privacy-cookie-policy/"},{"id":17,"name":"Greenhouse Group BV (with its trademark LemonPI)","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.lemonpi.io/privacy-policy/","deletedDate":"2019-08-06T00:00:00Z"},{"id":61,"name":"GumGum, Inc.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://gumgum.com/privacy-policy"},{"id":40,"name":"Active Agent (ADITION technologies AG)","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"http://www.active-agent.com/de/unternehmen/datenschutzerklaerung/"},{"id":76,"name":"PubMatic, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[],"policyUrl":"https://pubmatic.com/privacy-policy/"},{"id":89,"name":"Tapad, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[2],"policyUrl":"https://www.tapad.com/eu-privacy-policy"},{"id":46,"name":"Skimbit Ltd","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"https://skimlinks.com/pages/privacy-policy"},{"id":66,"name":"adsquare GmbH","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.adsquare.com/privacy"},{"id":105,"name":"Impression Desk Technologies Limited","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://impressiondesk.com/privacy-policy/","deletedDate":"2019-08-06T00:00:00Z"},{"id":41,"name":"Adverline","purposeIds":[2],"legIntPurposeIds":[1,3],"featureIds":[],"policyUrl":"https://www.adverline.com/privacy/"},{"id":82,"name":"Smaato, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.smaato.com/privacy/"},{"id":60,"name":"Rakuten Marketing LLC","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[1,2],"policyUrl":"https://rakutenadvertising.com/legal-notices/services-privacy-policy/"},{"id":70,"name":"Yieldlab AG","purposeIds":[],"legIntPurposeIds":[1,3],"featureIds":[3],"policyUrl":"http://www.yieldlab.de/meta-navigation/datenschutz/"},{"id":50,"name":"Adform","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://site.adform.com/privacy-center/platform-privacy/product-and-services-privacy-policy/"},{"id":48,"name":"NetSuccess, s.r.o.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.inres.sk/pp/"},{"id":100,"name":"Fifty Technology Limited","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://fifty.io/privacy-policy.php"},{"id":21,"name":"The Trade Desk","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[1,2,3],"policyUrl":"https://www.thetradedesk.com/general/privacy-policy"},{"id":110,"name":"Dynata LLC","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.opinionoutpost.co.uk/en-gb/policies/privacy"},{"id":42,"name":"Taboola Europe Limited","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://www.taboola.com/privacy-policy"},{"id":112,"name":"Maytrics GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://maytrics.com/privacy.php","deletedDate":"2019-09-17T00:00:00Z"},{"id":77,"name":"comScore, Inc.","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.scorecardresearch.com/privacy.aspx?newlanguage=1"},{"id":109,"name":"LoopMe Limited","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://loopme.com/privacy-policy/"},{"id":120,"name":"Eyeota Pte Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://www.eyeota.com/privacy-center"},{"id":93,"name":"Adloox SA","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"http://adloox.com/disclaimer"},{"id":132,"name":"Teads ","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[],"policyUrl":"https://www.teads.com/privacy-policy/"},{"id":22,"name":"admetrics GmbH","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://admetrics.io/en/privacy_policy/"},{"id":102,"name":"Telaria SAS","purposeIds":[1,2],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://telaria.com/privacy-policy/"},{"id":108,"name":"Rich Audience Technologies SL","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://richaudience.com/privacy/"},{"id":18,"name":"Widespace AB","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.widespace.com/legal/privacy-policy-notice/"},{"id":122,"name":"Avid Media Ltd","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.avidglobalmedia.eu/privacy-policy.html"},{"id":97,"name":"LiveRamp, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.liveramp.com/service-privacy-policy/"},{"id":138,"name":"ConnectAd Realtime GmbH","purposeIds":[1,2],"legIntPurposeIds":[3,4],"featureIds":[],"policyUrl":"http://connectadrealtime.com/privacy/"},{"id":72,"name":"Nano Interactive GmbH","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.nanointeractive.com/privacy"},{"id":127,"name":"PIXIMEDIA SAS","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"https://piximedia.com/privacy/"},{"id":136,"name":"Str\u00f6er SSP GmbH (SSP)","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[2,3],"policyUrl":"https://www.stroeer.de/fileadmin/de/Konvergenz_und_Konzepte/Daten_und_Technologien/Stroeer_SSP/Downloads/Datenschutz_Stroeer_SSP.pdf"},{"id":111,"name":"Showheroes SE","purposeIds":[2,3],"legIntPurposeIds":[1,4,5],"featureIds":[],"policyUrl":"https://showheroes.com/privacy/"},{"id":56,"name":"Confiant Inc.","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.confiant.com/privacy","deletedDate":"2020-05-18T00:00:00Z"},{"id":124,"name":"Teemo SA","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://teemo.co/fr/confidentialite/"},{"id":154,"name":"YOC AG","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"https://yoc.com/privacy/"},{"id":38,"name":"Beemray Oy","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.beemray.com/privacy-policy/","deletedDate":"2020-06-19T00:00:00Z"},{"id":101,"name":"MiQ","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://wearemiq.com/privacy-policy/"},{"id":149,"name":"ADman Interactive SLU","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[2,3],"policyUrl":"https://admanmedia.com/politica.html?setLng=es"},{"id":151,"name":"Admedo Ltd","purposeIds":[],"legIntPurposeIds":[1,2,3],"featureIds":[3],"policyUrl":"https://www.admedo.com/privacy-policy","deletedDate":"2020-07-17T00:00:00Z"},{"id":153,"name":"MADVERTISE MEDIA","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://madvertise.com/en/gdpr/"},{"id":159,"name":"Underdog Media LLC ","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[],"policyUrl":"https://underdogmedia.com/privacy-policy/"},{"id":157,"name":"Seedtag Advertising S.L","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.seedtag.com/en/privacy-policy/"},{"id":145,"name":"Snapsort Inc., operating as Sortable","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[3],"policyUrl":"https://help.sortable.com/help/privacy-policy"},{"id":131,"name":"ID5 Technology SAS","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.id5.io/privacy"},{"id":158,"name":"Reveal Mobile, Inc","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://revealmobile.com/privacy"},{"id":147,"name":"Adacado Technologies Inc. (DBA Adacado)","purposeIds":[1,2,3,4],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.adacado.com/privacy-policy-april-25-2018/"},{"id":130,"name":"NextRoll, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.nextroll.com/privacy"},{"id":129,"name":"IPONWEB GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.iponweb.com/privacy-policy/"},{"id":128,"name":"BIDSWITCH GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.bidswitch.com/privacy-policy/"},{"id":168,"name":"EASYmedia GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://login.rtbmarket.com/gdpr"},{"id":164,"name":"Outbrain UK Ltd","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.outbrain.com/legal/privacy#privacy-policy"},{"id":144,"name":"district m inc.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://districtm.net/en/page/platforms-data-and-privacy-policy/"},{"id":163,"name":"Bombora Inc.","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2],"policyUrl":"https://bombora.com/privacy"},{"id":173,"name":"Yieldmo, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.yieldmo.com/privacy/"},{"id":88,"name":"TreSensa, Inc.","purposeIds":[1,3],"legIntPurposeIds":[2,5],"featureIds":[1],"policyUrl":"https://www.tresensa.com/eu-privacy"},{"id":78,"name":"Flashtalking, Inc.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.flashtalking.com/privacypolicy/"},{"id":59,"name":"Sift Media, Inc","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.sift.co/privacy"},{"id":114,"name":"Sublime","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"http://ayads.co/privacy.php"},{"id":175,"name":"FORTVISION","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://fortvision.com/POC/index.html","deletedDate":"2019-08-09T00:00:00Z"},{"id":133,"name":"digitalAudience","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://digitalaudience.io/legal/privacy-cookies/"},{"id":14,"name":"Adkernel LLC","purposeIds":[1,4],"legIntPurposeIds":[2,3,5],"featureIds":[1,3],"policyUrl":"http://adkernel.com/privacy-policy/"},{"id":180,"name":"Thirdpresence Oy","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[3],"policyUrl":"http://www.thirdpresence.com/privacy","deletedDate":"2019-10-02T00:00:00Z"},{"id":183,"name":"EMX Digital LLC","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[2,3],"policyUrl":"https://emxdigital.com/privacy/"},{"id":58,"name":"33Across","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://www.33across.com/privacy-policy"},{"id":140,"name":"Platform161","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://platform161.com/cookie-and-privacy-policy/"},{"id":90,"name":"Teroa S.A.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[],"policyUrl":"https://www.e-planning.net/en/privacy.html"},{"id":141,"name":"1020, Inc. dba Placecast and Ericsson Emodo","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[3],"policyUrl":"https://www.emodoinc.com/privacy-policy/"},{"id":142,"name":"Media.net Advertising FZ-LLC","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"https://www.media.net/en/privacy-policy"},{"id":209,"name":"Delta Projects AB","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[3],"policyUrl":"https://deltaprojects.com/data-collection-policy"},{"id":195,"name":"advanced store GmbH","purposeIds":[2,3],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"http://www.advanced-store.com/de/datenschutz/"},{"id":190,"name":"video intelligence AG","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.vi.ai/privacy-policy/"},{"id":84,"name":"Semasio GmbH","purposeIds":[],"legIntPurposeIds":[1,2,4,5],"featureIds":[],"policyUrl":"http://www.semasio.com/privacy-policy/"},{"id":65,"name":"Location Sciences AI Ltd","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.locationsciences.ai/privacy-policy/"},{"id":210,"name":"Zemanta, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[1],"policyUrl":"http://www.zemanta.com/legal/privacy"},{"id":200,"name":"Tapjoy, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.tapjoy.com/legal/#privacy-policy"},{"id":188,"name":"Sellpoints Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://retargeter.com/service-privacy-policy/","deletedDate":"2019-09-17T00:00:00Z"},{"id":217,"name":"2KDirect, Inc. (dba iPromote)","purposeIds":[],"legIntPurposeIds":[1,3],"featureIds":[],"policyUrl":"https://www.ipromote.com/privacy-policy/"},{"id":156,"name":"Centro, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1],"policyUrl":"https://www.centro.net/privacy-policy/"},{"id":194,"name":"Rezonence Limited","purposeIds":[3,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://rezonence.com/privacy-policy/"},{"id":226,"name":"Publicis Media GmbH","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1],"policyUrl":"https://www.publicismedia.de/datenschutz/"},{"id":198,"name":"SYNC","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://redirect.sync.tv/privacy/","deletedDate":"2019-09-12T00:00:00Z"},{"id":227,"name":"ORTEC B.V.","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.ortecadscience.com/privacy-policy/"},{"id":225,"name":"Ligatus GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[3],"policyUrl":"https://www.ligatus.com/en/privacy-policy","deletedDate":"2020-06-19T00:00:00Z"},{"id":205,"name":"Adssets AB","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"http://adssets.com/policy/"},{"id":179,"name":"Collective Europe Ltd.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.collectiveuk.com/privacy.html"},{"id":31,"name":"Ogury Ltd.","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[2,3],"policyUrl":"https://www.ogury.com/privacy-policy/"},{"id":92,"name":"1plusX AG","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.1plusx.com/privacy-policy/"},{"id":155,"name":"AntVoice","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.antvoice.com/en/privacypolicy/"},{"id":115,"name":"smartclip Europe GmbH","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[2],"policyUrl":"https://privacy-portal.smartclip.net/"},{"id":126,"name":"DoubleVerify Inc.\u200b","purposeIds":[5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.doubleverify.com/privacy/"},{"id":193,"name":"Mediasmart Mobile S.L.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://mediasmart.io/privacy/"},{"id":245,"name":"IgnitionOne","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2],"policyUrl":"https://www.ignitionone.com/privacy-policy/","deletedDate":"2020-06-30T00:00:00Z"},{"id":213,"name":"emetriq GmbH","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://www.emetriq.com/datenschutz/"},{"id":244,"name":"Temelio","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://temelio.com/vie-privee"},{"id":224,"name":"adrule mobile GmbH","purposeIds":[2,4],"legIntPurposeIds":[1,3,5],"featureIds":[3],"policyUrl":"https://www.adrule.net/de/datenschutz/"},{"id":174,"name":"A Million Ads Ltd","purposeIds":[2],"legIntPurposeIds":[1,3],"featureIds":[],"policyUrl":"https://www.amillionads.com/privacy-policy"},{"id":192,"name":"remerge GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://remerge.io/privacy-policy.html"},{"id":232,"name":"Rockerbox, Inc","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[3],"policyUrl":"http://rockerbox.com/privacy","deletedDate":"2020-07-17T00:00:00Z"},{"id":256,"name":"Bounce Exchange, Inc","purposeIds":[1],"legIntPurposeIds":[2,4,5],"featureIds":[1,2],"policyUrl":"https://www.bouncex.com/privacy/"},{"id":234,"name":"ZBO Media","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[1,2],"policyUrl":"https://zbo.media/mentions-legales/politique-de-confidentialite-service-publicitaire/"},{"id":246,"name":"Smartology Limited","purposeIds":[3],"legIntPurposeIds":[4,5],"featureIds":[],"policyUrl":"https://www.smartology.net/privacy-policy/"},{"id":241,"name":"OneTag Limited","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://www.onetag.com/privacy/"},{"id":254,"name":"LiquidM Technology GmbH","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://liquidm.com/privacy-policy/"},{"id":215,"name":"ARMIS SAS","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://armis.tech/en/armis-personal-data-privacy-policy/"},{"id":167,"name":"Audiens S.r.l.","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.audiens.com/privacy"},{"id":240,"name":"7Hops.com Inc. (ZergNet)","purposeIds":[],"legIntPurposeIds":[1,4,5],"featureIds":[],"policyUrl":"https://zergnet.com/privacy"},{"id":235,"name":"Bucksense Inc","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[2,3],"policyUrl":"http://www.bucksense.com/platform-privacy-policy/"},{"id":185,"name":"Bidtellect, Inc","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.bidtellect.com/privacy-policy/"},{"id":258,"name":"Adello Group AG","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[3],"policyUrl":"https://www.adello.com/privacy-policy/"},{"id":169,"name":"RTK.IO, Inc","purposeIds":[1,4],"legIntPurposeIds":[2,3,5],"featureIds":[1,3],"policyUrl":"http://www.rtk.io/privacy.html","deletedDate":"2020-07-17T00:00:00Z"},{"id":208,"name":"Spotad","purposeIds":[2,3],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.spotad.co/privacy-policy/","deletedDate":"2020-07-06T00:00:00Z"},{"id":211,"name":"AdTheorent, Inc","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://adtheorent.com/privacy-policy"},{"id":229,"name":"Digitize New Media Ltd","purposeIds":[2,4],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.digitize.ie/online-privacy","deletedDate":"2020-07-17T00:00:00Z"},{"id":273,"name":"Bannerflow AB","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://www.bannerflow.com/privacy "},{"id":104,"name":"Sonobi, Inc","purposeIds":[1,2],"legIntPurposeIds":[3],"featureIds":[1,2,3],"policyUrl":"http://sonobi.com/privacy-policy/"},{"id":162,"name":"Unruly Group Ltd","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[1,2],"policyUrl":"https://unruly.co/privacy/"},{"id":249,"name":"Spolecznosci Sp. z o.o. Sp. k.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.spolecznosci.pl/polityka-prywatnosci"},{"id":125,"name":"Research Now Group, Inc","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.valuedopinions.co.uk/privacy","deletedDate":"2019-09-17T00:00:00Z"},{"id":170,"name":"Goodway Group, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://goodwaygroup.com/privacy-policy/"},{"id":160,"name":"Netsprint SA","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://netsprint.eu/privacy.html"},{"id":189,"name":"Intowow Innovation Ltd.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://www.intowow.com/privacy/","deletedDate":"2019-08-12T00:00:00Z"},{"id":279,"name":"Mirando GmbH & Co KG","purposeIds":[],"legIntPurposeIds":[1],"featureIds":[3],"policyUrl":"https://wwwmirando.de/datenschutz/"},{"id":269,"name":"Sanoma Media Finland","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://sanoma.fi/tietoa-meista/tietosuoja/","deletedDate":"2019-08-07T00:00:00Z"},{"id":276,"name":"Viralize SRL","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"https://viralize.com/privacy-policy"},{"id":87,"name":"Genius Sports Media Limited","purposeIds":[2,4],"legIntPurposeIds":[1,3,5],"featureIds":[2,3],"policyUrl":"https://www.geniussports.com/privacy-policy"},{"id":182,"name":"Collective, Inc. dba Visto","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.vistohub.com/privacy-policy/","deletedDate":"2019-07-26T00:00:00Z"},{"id":255,"name":"Onnetwork Sp. z o.o.","purposeIds":[2,3,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://www.onnetwork.tv/pp_services.php"},{"id":203,"name":"Revcontent, LLC","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://intercom.help/revcontent2/en/articles/2290675-revcontent-s-privacy-policy"},{"id":260,"name":"RockYou, Inc.","purposeIds":[3],"legIntPurposeIds":[1,2,5],"featureIds":[3],"policyUrl":"https://rockyou.com/privacy-policy/","deletedDate":"2019-08-09T00:00:00Z"},{"id":237,"name":"LKQD, a division of Nexstar Digital, LLC.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[2,3],"policyUrl":"http://www.lkqd.com/privacy-policy/","deletedDate":"2019-09-12T00:00:00Z"},{"id":274,"name":"Golden Bees","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.goldenbees.fr/en/privacy-charter/"},{"id":280,"name":"Spot.IM LTD","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.spot.im/privacy/"},{"id":239,"name":"Triton Digital Canada Inc.","purposeIds":[],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"https://www.tritondigital.com/privacy-policies"},{"id":177,"name":"plista GmbH","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.plista.com/about/privacy/"},{"id":201,"name":"TimeOne","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://privacy.timeonegroup.com/en/","deletedDate":"2020-05-15T00:00:00Z"},{"id":150,"name":"Inskin Media LTD","purposeIds":[2,3,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"http://www.inskinmedia.com/privacy-policy.html"},{"id":252,"name":"Jaduda GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[3],"policyUrl":"https://www.jadudamobile.com/datenschutzerklaerung/"},{"id":248,"name":"Converge-Digital","purposeIds":[1],"legIntPurposeIds":[3,4,5],"featureIds":[3],"policyUrl":"https://converge-digital.com/privacy-policy/"},{"id":161,"name":"Smadex SL","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://smadex.com/end-user-privacy-policy/"},{"id":285,"name":"Comcast International France SAS","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.freewheel.com/privacy-policy"},{"id":228,"name":"McCann Discipline LTD","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.primis.tech/privacy-policy/"},{"id":299,"name":"AdClear GmbH","purposeIds":[1,5],"legIntPurposeIds":[2,3,4],"featureIds":[1,2],"policyUrl":"https://www.adclear.de/datenschutzerklaerung/"},{"id":277,"name":"Codewise VL Sp. z o.o. Sp. k","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://voluumdsp.com/end-user-privacy-policy/"},{"id":259,"name":"ADYOULIKE SA","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://www.adyoulike.com/privacy_policy.php"},{"id":272,"name":"A.Mob","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.we-are-adot.com/privacy-policy/"},{"id":230,"name":"Steel House, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://steelhouse.com/privacy-policy/"},{"id":253,"name":"Improve Digital BV","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.improvedigital.com/platform-privacy-policy"},{"id":304,"name":"On Device Research Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://s.on-device.com/privacyPolicy"},{"id":314,"name":"Keymantics","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.keymantics.com/assets/privacy-policy.pdf"},{"id":257,"name":"R-TARGET","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"http://www.r-target.com/privacy","deletedDate":"2019-08-26T00:00:00Z"},{"id":317,"name":"mainADV Srl","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"http://www.mainad.com/privacy-policy/"},{"id":278,"name":"Integral Ad Science, Inc.","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://integralads.com/privacy-policy/"},{"id":291,"name":"Qwertize","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://www.qwertize.com/en/privacy","deletedDate":"2019-08-26T00:00:00Z"},{"id":295,"name":"Sojern, Inc.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.sojern.com/privacy/product-privacy-policy/"},{"id":315,"name":"Celtra, Inc.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.celtra.com/privacy-policy/"},{"id":165,"name":"SpotX, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1],"policyUrl":"https://www.spotx.tv/privacy-policy/"},{"id":47,"name":"ADMAN - Phaistos Networks, S.A.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.adman.gr/privacy"},{"id":134,"name":"SMARTSTREAM.TV GmbH","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[2],"policyUrl":"https://www.smartstream.tv/en/productprivacy"},{"id":325,"name":"Knorex","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.knorex.com/privacy"},{"id":316,"name":"Gamned","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.gamned.com/privacy-policy/"},{"id":318,"name":"Accorp Sp. z o.o.","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"http://www.instytut-pollster.pl/privacy-policy/"},{"id":199,"name":"ADUX","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.adux.com/donnees-personelles/"},{"id":236,"name":"PowerLinks Media Limited","purposeIds":[1,2,5],"legIntPurposeIds":[3,4],"featureIds":[3],"policyUrl":"https://www.powerlinks.com/privacy-policy/"},{"id":294,"name":"Jivox Corporation","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.jivox.com/privacy"},{"id":143,"name":"Connatix Native Exchange Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://connatix.com/privacy-policy/"},{"id":297,"name":"Polar Mobile Group Inc.","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[],"policyUrl":"https://privacy.polar.me"},{"id":319,"name":"Clipcentric, Inc.","purposeIds":[1,2],"legIntPurposeIds":[3],"featureIds":[3],"policyUrl":"https://clipcentric.com/privacy.bhtml"},{"id":290,"name":"Readpeak Oy","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://readpeak.com/privacy-policy/"},{"id":323,"name":"DAZN Media Services Limited","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://www.goal.com/en-gb/legal/privacy-policy"},{"id":119,"name":"Fusio by S4M","purposeIds":[1,2,5],"legIntPurposeIds":[3],"featureIds":[1,3],"policyUrl":"http://www.s4m.io/privacy-policy/"},{"id":302,"name":"Mobile Professionals BV","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://mobpro.com/privacy.html"},{"id":212,"name":"usemax advertisement (Emego GmbH)","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"http://www.usemax.de/?l=privacy"},{"id":264,"name":"Adobe Advertising Cloud","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.adobe.com/privacy/experience-cloud.html"},{"id":44,"name":"The ADEX GmbH","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://theadex.com/privacy-opt-out/"},{"id":282,"name":"Welect GmbH","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.welect.de/datenschutz"},{"id":238,"name":"StackAdapt","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.stackadapt.com/privacy"},{"id":284,"name":"WEBORAMA","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://weborama.com/privacy_en/"},{"id":148,"name":"Liveintent Inc.","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://liveintent.com/services-privacy-policy/"},{"id":64,"name":"DigiTrust / IAB Tech Lab","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.digitru.st/privacy-policy/"},{"id":301,"name":"zeotap GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://zeotap.com/privacy_policy"},{"id":275,"name":"TabMo SAS","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"http://static.tabmo.io.s3.amazonaws.com/privacy-policy/index.html"},{"id":310,"name":"Adevinta Spain S.L.U.","purposeIds":[],"legIntPurposeIds":[4],"featureIds":[],"policyUrl":"https://www.adevinta.com/about/privacy/"},{"id":139,"name":"Permodo GmbH","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://permodo.com/de/privacy.html"},{"id":326,"name":"AdTiming Technology Company Limited","purposeIds":[3,5],"legIntPurposeIds":[1,2,4],"featureIds":[],"policyUrl":"http://www.adtiming.com/en/privacypolicy.html"},{"id":262,"name":"Fyber ","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.fyber.com/legal/privacy-policy/"},{"id":331,"name":"ad6media","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[2,3],"policyUrl":"https://www.ad6media.fr/privacy"},{"id":345,"name":"The Kantar Group Limited","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.kantar.com/cookies-policies"},{"id":308,"name":"Rockabox Media Ltd","purposeIds":[1],"legIntPurposeIds":[2,3],"featureIds":[],"policyUrl":"http://scoota.com/privacy-policy"},{"id":270,"name":"Marfeel Solutions, SL","purposeIds":[],"legIntPurposeIds":[2],"featureIds":[],"policyUrl":"https://www.marfeel.com/privacy-policy/"},{"id":333,"name":"InMobi Pte Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.inmobi.com/privacy-policy-for-eea"},{"id":202,"name":"Telaria, Inc","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://telaria.com/privacy-policy/"},{"id":328,"name":"Gemius SA","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.gemius.com/cookie-policy.html"},{"id":281,"name":"Wizaly","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.wizaly.com/terms-of-use#privacy-policy"},{"id":354,"name":"Apester Ltd","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"https://apester.com/privacy-policy/"},{"id":320,"name":"Adelphic LLC","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://adelphic.com/platform/privacy/"},{"id":359,"name":"AerServ LLC","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://www.inmobi.com/privacy-policy-for-eea"},{"id":265,"name":"Instinctive, Inc.","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"https://instinctive.io/privacy"},{"id":349,"name":"Optomaton UG","purposeIds":[1,2,3,4],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://optomaton.com/privacy.html"},{"id":288,"name":"Video Media Groep B.V.","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"http://www.videomediagroup.com/wp-content/uploads/2016/01/Privacy-policy-VMG.pdf","deletedDate":"2019-09-17T00:00:00Z"},{"id":266,"name":"Digilant Spain, SLU","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://www.digilant.com/es/politica-privacidad/"},{"id":339,"name":"Vuble","purposeIds":[2,3,4,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://www.vuble.tv/us/privacy","deletedDate":"2019-08-26T00:00:00Z"},{"id":303,"name":"Orion Semantics","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://static.orion-semantics.com/privacy.html"},{"id":261,"name":"Signal Digital Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.signal.co/privacy-policy/"},{"id":83,"name":"Visarity Technologies GmbH","purposeIds":[5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://primo.design/docs/PrivacyPolicyPrimo.html"},{"id":343,"name":"DIGITEKA Technologies","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[3],"policyUrl":"https://www.ultimedia.com/POLICY.html"},{"id":330,"name":"Linicom","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"https://www.linicom.com/privacy/","deletedDate":"2020-06-08T00:00:00Z"},{"id":231,"name":"AcuityAds Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://privacy.acuityads.com/corporate-privacy-policy.html"},{"id":216,"name":"Mindlytix SAS","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"http://mindlytix.com/privacy/"},{"id":311,"name":"Mobfox US LLC","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.mobfox.com/privacy-policy/"},{"id":358,"name":"MGID Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.mgid.com/privacy-policy"},{"id":152,"name":"Meetrics GmbH","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.meetrics.com/en/data-privacy/"},{"id":251,"name":"Yieldlove GmbH","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[],"policyUrl":"http://www.yieldlove.com/cookie-policy"},{"id":344,"name":"My6sense Inc.","purposeIds":[1,3,5],"legIntPurposeIds":[2,4],"featureIds":[],"policyUrl":"https://my6sense.com/privacy-policy/","deletedDate":"2020-07-17T00:00:00Z"},{"id":347,"name":"Ezoic Inc.","purposeIds":[2,4,5],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://www.ezoic.com/terms/"},{"id":218,"name":"Bigabid Media ltd","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"https://www.bigabid.com/privacy-policy"},{"id":350,"name":"Free Stream Media Corp. dba Samba TV","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://samba.tv/legal/privacy-policy/"},{"id":351,"name":"Samba TV UK Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://samba.tv/legal/privacy-policy/"},{"id":341,"name":"Somo Audience Corp","purposeIds":[1,2],"legIntPurposeIds":[3],"featureIds":[1,2,3],"policyUrl":"https://somoaudience.com/legal/","deletedDate":"2020-07-06T00:00:00Z"},{"id":380,"name":"Vidoomy Media SL","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[2,3],"policyUrl":"http://vidoomy.com/privacy-policy.html"},{"id":378,"name":"communicationAds GmbH & Co. KG","purposeIds":[],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://www.communicationads.net/aboutus/privacy/"},{"id":369,"name":"Getintent USA, inc.","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://getintent.com/privacy/","deletedDate":"2020-07-17T00:00:00Z"},{"id":184,"name":"mediarithmics SAS","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.mediarithmics.com/en-us/content/privacy-policy"},{"id":368,"name":"VECTAURY","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.vectaury.io/en/personal-data"},{"id":373,"name":"Nielsen Marketing Cloud","purposeIds":[1,2],"legIntPurposeIds":[5],"featureIds":[1,2],"policyUrl":"http://www.nielsen.com/us/en/privacy-statement/exelate-privacy-policy.html"},{"id":214,"name":"Digital Control GmbH & Co. KG","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://advolution.de/privacy.php","deletedDate":"2020-05-06T00:00:00Z"},{"id":388,"name":"numberly","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://numberly.com/en/privacy/"},{"id":250,"name":"Qriously Ltd","purposeIds":[2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.brandwatch.com/legal/qriously-privacy-notice/"},{"id":223,"name":"Audience Trading Platform Ltd.","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[2],"policyUrl":"https://atp.io/privacy-policy"},{"id":387,"name":"Triapodi Ltd.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://appreciate.mobi/page.html#/end-user-privacy-policy"},{"id":312,"name":"Exactag GmbH","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.exactag.com/en/data-privacy/"},{"id":178,"name":"Hybrid Theory","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://hybridtheory.com/privacy-policy/"},{"id":377,"name":"AddApptr GmbH","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.addapptr.com/data-privacy"},{"id":382,"name":"The Reach Group GmbH","purposeIds":[1,2,4,5],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://trg.de/en/privacy-statement/"},{"id":206,"name":"Hybrid Adtech GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://hybrid.ai/data_protection_policy"},{"id":403,"name":"Mobusi Mobile Advertising S.L.","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.mobusi.com/privacy.en.html","deletedDate":"2020-07-17T00:00:00Z"},{"id":385,"name":"Oracle Data Cloud","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"https://www.oracle.com/legal/privacy/marketing-cloud-data-cloud-privacy-policy.html"},{"id":404,"name":"Duplo Media AS","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"https://www.easy-ads.com/privacypolicy.htm"},{"id":242,"name":"twiago GmbH","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.twiago.com/datenschutz/"},{"id":376,"name":"Pocketmath Pte Ltd","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.pocketmath.com/privacy-policy"},{"id":402,"name":"Effiliation","purposeIds":[2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://inter.effiliation.com/politique-confidentialite.html"},{"id":413,"name":"Eulerian Technologies","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.eulerian.com/en/privacy/"},{"id":400,"name":"Whenever Media Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.whenevermedia.com/privacy-policy","deletedDate":"2019-07-29T00:00:00Z"},{"id":171,"name":"Webedia","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.webedia-group.com/site/privacy-policy","deletedDate":"2020-07-01T00:00:00Z"},{"id":398,"name":"Yormedia Solutions Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://www.yormedia.com/privacy-and-cookies-notice/","deletedDate":"2019-08-06T00:00:00Z"},{"id":415,"name":"Seenthis AB","purposeIds":[],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://seenthis.co/privacy-notice-2018-04-18.pdf"},{"id":263,"name":"Nativo, Inc.","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.nativo.com/interest-based-ads"},{"id":329,"name":"Browsi Mobile Ltd","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://gobrowsi.com/browsi-privacy-policy/"},{"id":389,"name":"Bidmanagement GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.adspert.net/en/privacy/","deletedDate":"2020-07-01T00:00:00Z"},{"id":337,"name":"SheMedia, LLC","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.shemedia.com/ad-services-privacy-policy"},{"id":422,"name":"Brand Metrics Sweden AB","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://collector.brandmetrics.com/brandmetrics_privacypolicy.pdf"},{"id":421,"name":"LeftsnRight, Inc. dba LIQWID","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://liqwid.solutions/privacy-policy","deletedDate":"2020-06-30T00:00:00Z"},{"id":426,"name":"TradeTracker","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[2],"policyUrl":"https://tradetracker.com/privacy-policy/","deletedDate":"2019-08-21T00:00:00Z"},{"id":394,"name":"AudienceProject Aps","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2],"policyUrl":"https://privacy.audienceproject.com"},{"id":287,"name":"Avazu Inc.","purposeIds":[],"legIntPurposeIds":[1,3,4],"featureIds":[3],"policyUrl":"http://avazuinc.com/opt-out/","deletedDate":"2020-08-03T00:00:00Z"},{"id":243,"name":"Cloud Technologies S.A.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.cloudtechnologies.pl/en/internet-advertising-privacy-policy"},{"id":113,"name":"iotec global Ltd.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.iotecglobal.com/privacy-policy/","deletedDate":"2020-07-06T00:00:00Z"},{"id":338,"name":"dunnhumby Germany GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.sociomantic.com/privacy/en/","deletedDate":"2020-07-17T00:00:00Z"},{"id":405,"name":"IgnitionAi Ltd","purposeIds":[],"legIntPurposeIds":[1,3,4,5],"featureIds":[2],"policyUrl":"https://www.isitelab.io/default.aspx","deletedDate":"2020-07-03T00:00:00Z"},{"id":416,"name":"Commanders Act","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.commandersact.com/en/privacy/"},{"id":434,"name":"DynAdmic","purposeIds":[1,3],"legIntPurposeIds":[2,4],"featureIds":[1,3],"policyUrl":"http://eu.dynadmic.com/privacy-policy/"},{"id":435,"name":"SINGLESPOT SAS ","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"https://www.singlespot.com/privacy_policy?locale=fr"},{"id":409,"name":"Arrivalist Co.","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[1,2],"policyUrl":"https://www.arrivalist.com/privacy"},{"id":321,"name":"Ziff Davis LLC","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.ziffdavis.com/privacy-policy"},{"id":436,"name":"INVIBES GROUP","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[1,2,3],"policyUrl":"http://www.invibes.com/terms"},{"id":442,"name":"R-Advertising","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.tradedoubler.com/en/privacy-policy/","deletedDate":"2019-08-20T00:00:00Z"},{"id":362,"name":"Myntelligence S.r.l.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://myntelligence.com/privacy-page/"},{"id":418,"name":"PROXISTORE","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[3],"policyUrl":"https://www.proxistore.com/common/en/cgv"},{"id":449,"name":"Mobile Journey B.V.","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[3],"policyUrl":"https://mobilejourney.com/Privacy-Policy","deletedDate":"2019-09-05T00:00:00Z"},{"id":443,"name":"Tradedoubler AB","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[2],"policyUrl":"https://www.tradedoubler.com/en/privacy-policy/","deletedDate":"2019-08-13T00:00:00Z"},{"id":429,"name":"Signals","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://signalsdata.com/platform-cookie-policy/"},{"id":335,"name":"Beachfront Media LLC","purposeIds":[],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"http://beachfront.com/privacy-policy/"},{"id":407,"name":"Publishers Internationale Pty Ltd","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.pi-rate.com.au/privacy.html","deletedDate":"2019-11-08T00:00:00Z"},{"id":427,"name":"Proxi.cloud Sp. z o.o.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://proxi.cloud/info/privacy-policy/"},{"id":374,"name":"Bmind a Sales Maker Company, S.L.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.bmind.es/legal-notice/"},{"id":438,"name":"INVIDI technologies AB","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"http://www.invidi.com/wp-content/uploads/2020/02/ad-tech-services-privacy-policy.pdf"},{"id":450,"name":"Neodata Group srl","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.neodatagroup.com/en/security-policy"},{"id":452,"name":"Innovid Inc.","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"http://www.innovid.com/privacy-policy"},{"id":444,"name":"Playbuzz Ltd (aka EX.CO)","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"https://ex.co/privacy-policy/"},{"id":412,"name":"Cxense ASA","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.cxense.com/about-us/privacy-policy"},{"id":454,"name":"Adimo","purposeIds":[1],"legIntPurposeIds":[5],"featureIds":[3],"policyUrl":"https://adimo.co/privacy-policy/","deletedDate":"2019-09-12T00:00:00Z"},{"id":455,"name":"GDMServices, Inc. d/b/a FiksuDSP","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://fiksu.com/privacy-policy/"},{"id":298,"name":"Cuebiq Inc.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.cuebiq.com/privacypolicy/","deletedDate":"2019-08-30T00:00:00Z"},{"id":423,"name":"travel audience GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://travelaudience.com/product-privacy-policy/"},{"id":397,"name":"Demandbase, Inc. ","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.demandbase.com/privacy-policy/"},{"id":381,"name":"Solocal","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://frontend.adhslx.com/privacy.html?"},{"id":425,"name":"ADRINO Sp. z o.o.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"http://www.adrino.pl/ciasteczkowa-polityka/","deletedDate":"2019-09-05T00:00:00Z"},{"id":365,"name":"Forensiq LLC","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[1,3],"policyUrl":"https://impact.com/privacy-policy/"},{"id":447,"name":"Adludio Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.adludio.com/privacy-policy/"},{"id":410,"name":"Adtelligent Inc.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://adtelligent.com/privacy-policy/"},{"id":137,"name":"Str\u00f6er SSP GmbH (DSP)","purposeIds":[],"legIntPurposeIds":[1,2,3],"featureIds":[],"policyUrl":"https://www.stroeer.de/fileadmin/de/Konvergenz_und_Konzepte/Daten_und_Technologien/Stroeer_SSP/Downloads/Datenschutz_Stroeer_SSP.pdf"},{"id":395,"name":"PREX Programmatic Exchange GmbH&Co KG","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[],"policyUrl":"http://www.programmatic-exchange.com/privacy","deletedDate":"2020-07-03T00:00:00Z"},{"id":462,"name":"Bidstack Limited","purposeIds":[1,2,5],"legIntPurposeIds":[3,4],"featureIds":[2],"policyUrl":"https://www.bidstack.com/privacy-policy/"},{"id":466,"name":"TACTIC\u2122 Real-Time Marketing AS","purposeIds":[],"legIntPurposeIds":[4,5],"featureIds":[],"policyUrl":"https://tacticrealtime.com/privacy/"},{"id":340,"name":"Yieldr UK","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"https://www.yieldr.com/privacy"},{"id":336,"name":"Telecoming S.A.","purposeIds":[3,4],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"http://www.telecoming.com/privacy-policy/"},{"id":430,"name":"Ad Unity Ltd","purposeIds":[],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"http://www.adunity.com/privacy-policy.html","deletedDate":"2019-08-13T00:00:00Z"},{"id":346,"name":"Cybba, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://cybba.com/about/legal/data-processing-agreement/","deletedDate":"2020-08-03T00:00:00Z"},{"id":469,"name":"Zeta Global","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://zetaglobal.com/privacy-policy/"},{"id":440,"name":"DEFINE MEDIA GMBH","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"http://www.definemedia.de/datenschutz-conative/"},{"id":375,"name":"Affle International","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://affle.com/privacy-policy "},{"id":196,"name":"AdElement Media Solutions Pvt Ltd","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"http://adelement.com/privacy-policy.html"},{"id":268,"name":"Social Tokens Ltd. ","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://woobi.com/privacy/","deletedDate":"2019-10-02T00:00:00Z"},{"id":475,"name":"TAPTAP Digital SL","purposeIds":[1,2,3],"legIntPurposeIds":[4,5],"featureIds":[1,2,3],"policyUrl":"http://www.taptapnetworks.com/privacy_policy/"},{"id":474,"name":"hbfsTech","purposeIds":[],"legIntPurposeIds":[1,2],"featureIds":[],"policyUrl":"http://www.hbfstech.com/fr/privacy.html"},{"id":448,"name":"Targetspot Belgium SPRL","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://marketing.targetspot.com/Targetspot/Legal/TargetSpot%20Privacy%20Policy%20-%20June%202018.pdf"},{"id":428,"name":"Internet BillBoard a.s.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"http://www.ibillboard.com/en/privacy-information/"},{"id":461,"name":"B2B Media Group EMEA GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.selfcampaign.com/static/privacy","deletedDate":"2019-08-14T00:00:00Z"},{"id":476,"name":"HIRO Media Ltd","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1],"policyUrl":"http://hiro-media.com/privacy.php"},{"id":480,"name":"pilotx.tv","purposeIds":[2,3],"legIntPurposeIds":[1,4,5],"featureIds":[1,2,3],"policyUrl":"https://pilotx.tv/privacy/"},{"id":366,"name":"CerebroAd.com s.r.o.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.cerebroad.com/privacy-policy","deletedDate":"2019-10-02T00:00:00Z"},{"id":392,"name":"Str\u00f6er Mobile Performance GmbH","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[3],"policyUrl":"https://stroeermobileperformance.com/?dl=privacy"},{"id":357,"name":"Totaljobs Group Ltd ","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.totaljobs.com/privacy-policy"},{"id":486,"name":"Madington","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"https://delivered-by-madington.com/dat-privacy-policy/"},{"id":468,"name":"NeuStar, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,5],"featureIds":[1,2],"policyUrl":"https://www.home.neustar/privacy"},{"id":458,"name":"AdColony, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1],"policyUrl":"adcolony.com/privacy-policy/"},{"id":489,"name":"YellowHammer Media Group","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.yhmg.com/privacy-policy/","deletedDate":"2019-11-27T00:00:00Z"},{"id":293,"name":"SpringServe, LLC","purposeIds":[],"legIntPurposeIds":[1,3],"featureIds":[],"policyUrl":"https://springserve.com/privacy-policy/"},{"id":484,"name":"STRIATUM SAS","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://adledge.com/data-privacy/"},{"id":493,"name":"Carbon (AI) Limited","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"https://carbonrmp.com/privacy.html"},{"id":495,"name":"Arcspire Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://public.arcspire.io/privacy.pdf"},{"id":496,"name":"Automattic Inc.","purposeIds":[1,2,3,4],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://en.blog.wordpress.com/2017/12/04/updated-privacy-policy/"},{"id":424,"name":"KUPONA GmbH","purposeIds":[],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://www.kupona.de/dsgvo/"},{"id":408,"name":"Fidelity Media","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[3],"policyUrl":"https://fidelity-media.com/privacy-policy/"},{"id":473,"name":"Sub2 Technologies Ltd","purposeIds":[3,4,5],"legIntPurposeIds":[1,2],"featureIds":[],"policyUrl":"http://www.sub2tech.com/privacy-policy/"},{"id":467,"name":"Haensel AMS GmbH","purposeIds":[3,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://haensel-ams.com/data-privacy/"},{"id":490,"name":"PLAYGROUND XYZ EMEA LTD","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://playground.xyz/privacy"},{"id":464,"name":"Oracle AddThis","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[2],"policyUrl":"http://www.addthis.com/privacy/privacy-policy/","deletedDate":"2020-02-12T00:00:00Z"},{"id":491,"name":"Triboo Data Analytics","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.shinystat.com/it/informativa_privacy_generale.html"},{"id":499,"name":"PurposeLab, LLC","purposeIds":[5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://purposelab.com/privacy/","deletedDate":"2019-10-02T00:00:00Z"},{"id":502,"name":"NEXD","purposeIds":[5],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://nexd.com/privacy-policy"},{"id":465,"name":"Schibsted Product and Tech UK","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://www.schibsted.com/","deletedDate":"2019-07-26T00:00:00Z"},{"id":497,"name":"Little Big Data sp.z.o.o.","purposeIds":[1,2,4],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://dtxngr.com/legal/"},{"id":492,"name":"LotaData, Inc.","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[1],"policyUrl":"https://lotadata.com/privacy_policy","deletedDate":"2019-10-02T00:00:00Z"},{"id":512,"name":"PubNative GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://pubnative.net/privacy-notice/"},{"id":471,"name":"FlexOffers.com, LLC","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.flexoffers.com/privacy-policy/","deletedDate":"2019-11-18T00:00:00Z"},{"id":494,"name":"Cablato Limited","purposeIds":[1,2,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://cablato.com/privacy","deletedDate":"2019-10-02T00:00:00Z"},{"id":516,"name":"Pexi B.V.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://pexi.nl/privacy-policy/"},{"id":507,"name":"AdsWizz Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[1,2,3],"policyUrl":"https://www.adswizz.com/our-privacy-policy/"},{"id":482,"name":"UberMedia, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://ubermedia.com/summary-of-privacy-policy/"},{"id":505,"name":"Shopalyst Inc","purposeIds":[1,2],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.shortlyst.com/eu/privacy_terms.html"},{"id":517,"name":"SunMedia ","purposeIds":[1,2],"legIntPurposeIds":[3],"featureIds":[2],"policyUrl":"https://www.sunmedia.tv/en/cookies"},{"id":518,"name":"Accelerize Inc.","purposeIds":[1],"legIntPurposeIds":[5],"featureIds":[2,3],"policyUrl":"https://getcake.com/privacy-policy/","deletedDate":"2020-07-17T00:00:00Z"},{"id":511,"name":"Admixer EU GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://admixer.com/privacy/"},{"id":479,"name":"INFINIA MOBILE S.L.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.infiniamobile.com/privacy_policy"},{"id":513,"name":"Shopstyle","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.shopstyle.co.uk/privacy","deletedDate":"2019-10-02T00:00:00Z"},{"id":509,"name":"ATG Ad Tech Group GmbH","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://ad-tech-group.com/privacy-policy/"},{"id":521,"name":"netzeffekt GmbH","purposeIds":[1],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://www.netzeffekt.de/en/imprint"},{"id":487,"name":"nugg.ad GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[1],"policyUrl":"https://www.nugg.ad/en/privacy/general-information.html","deletedDate":"2019-10-03T00:00:00Z"},{"id":515,"name":"ZighZag","purposeIds":[1,3],"legIntPurposeIds":[5],"featureIds":[1,2],"policyUrl":"https://zighzag.com/privacy"},{"id":520,"name":"ChannelSight ","purposeIds":[1],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.channelsight.com/privacypolicy/"},{"id":524,"name":"The Ozone Project Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://ozoneproject.com/privacy-policy"},{"id":529,"name":"Fidzup","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.fidzup.com/en/privacy/","deletedDate":"2019-11-18T00:00:00Z"},{"id":528,"name":"Kayzen","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://kayzen.io/data-privacy-policy"},{"id":527,"name":"Jampp LTD","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://jampp.com/privacy.html"},{"id":506,"name":"salesforce.com, inc.","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.salesforce.com/company/privacy/"},{"id":534,"name":"SmartyAds Inc.","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://smartyads.com/privacy-policy"},{"id":535,"name":"INNITY","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.innity.com/privacy-policy.php"},{"id":514,"name":"Uprival LLC","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://uprival.com/privacy-policy/","deletedDate":"2020-01-21T00:00:00Z"},{"id":522,"name":"Tealium Inc.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://tealium.com/privacy-policy/"},{"id":530,"name":"Near Pte Ltd","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://near.co/privacy"},{"id":539,"name":"AdDefend GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.addefend.com/en/privacy-policy/"},{"id":501,"name":"Alliance Gravity Data Media","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.alliancegravity.com/politiquedeprotectiondesdonneespersonnelles"},{"id":519,"name":"Chargeads","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.chargeplatform.com/privacy"},{"id":523,"name":"X-Mode Social, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://xmode.io/privacy-policy.html"},{"id":537,"name":"RUN, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"http://www.runads.com/privacy-policy"},{"id":531,"name":"Smartclip Hispania SL","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://rgpd-smartclip.com/"},{"id":536,"name":"GlobalWebIndex","purposeIds":[1],"legIntPurposeIds":[3,5],"featureIds":[1,2],"policyUrl":"http://legal.trendstream.net/non-panellist_privacy_policy"},{"id":542,"name":"Densou Trading Desk ApS","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://densou.dk/Policy.html","deletedDate":"2020-01-21T00:00:00Z"},{"id":525,"name":"PUB OCEAN LIMITED","purposeIds":[1],"legIntPurposeIds":[5],"featureIds":[3],"policyUrl":"https://rta.pubocean.com/privacy-policy/","deletedDate":"2019-10-03T00:00:00Z"},{"id":544,"name":"Kochava Inc.","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[1,2],"policyUrl":"https://www.kochava.com/support-privacy/"},{"id":543,"name":"PaperG, Inc. dba Thunder Industries","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[],"policyUrl":"https://www.makethunder.com/privacy"},{"id":334,"name":"Cydersoft","purposeIds":[],"legIntPurposeIds":[1,2,3,4],"featureIds":[2,3],"policyUrl":"http://www.videmob.com/privacy.html"},{"id":551,"name":"Illuma Technology Limited","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://www.weareilluma.com/endddd","deletedDate":"2019-11-14T00:00:00Z"},{"id":540,"name":"Tunnl BV","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://tunnl.com/privacy.html","deletedDate":"2019-12-20T00:00:00Z"},{"id":547,"name":"Video Reach","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.videoreach.de/about/privacy-policy/","deletedDate":"2020-01-21T00:00:00Z"},{"id":546,"name":"Smart Traffik","purposeIds":[1],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://okube-attribution.com/politique-de-confidentialite/"},{"id":541,"name":"DeepIntent, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[3],"policyUrl":"https://www.deepintent.com/privacypolicy"},{"id":545,"name":"Reignn Platform Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://reignn.com/user-privacy-policy"},{"id":439,"name":"Bit Q Holdings Limited","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.rippll.com/privacy"},{"id":553,"name":"Adhese","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://adhese.com/privacy-and-cookie-policy"},{"id":556,"name":"adhood.com","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://v3.adhood.com/en/site/politikavekurallar/gizlilik.php?lang=en"},{"id":550,"name":"Happydemics","purposeIds":[5],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"https://www.iubenda.com/privacy-policy/69056167/full-legal"},{"id":560,"name":"Leiki Ltd.","purposeIds":[1,2,3],"legIntPurposeIds":[4],"featureIds":[],"policyUrl":"http://www.leiki.com/privacy","deletedDate":"2020-01-07T00:00:00Z"},{"id":554,"name":"RMSi Radio Marketing Service interactive GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.rms.de/datenschutz/"},{"id":498,"name":"Mediakeys Platform","purposeIds":[1],"legIntPurposeIds":[3],"featureIds":[3],"policyUrl":"https://drbanner.com/privacypolicy_en/"},{"id":565,"name":"Adobe Audience Manager","purposeIds":[1,2,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.adobe.com/privacy/policy.html"},{"id":118,"name":"Drawbridge, Inc.","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://www.drawbridge.com/privacy/","deletedDate":"2020-03-06T00:00:00Z"},{"id":572,"name":"CHEQ AI TECHNOLOGIES LTD.","purposeIds":[1],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"http://www.cheq.ai/privacy"},{"id":571,"name":"ViewPay","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://viewpay.tv/mentions-legales/"},{"id":568,"name":"Jointag S.r.l.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.jointag.com/privacy/kariboo/publisher/third/"},{"id":570,"name":"Czech Publisher Exchange z.s.p.o.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.cpex.cz/pro-uzivatele/ochrana-soukromi/"},{"id":559,"name":"Otto (GmbH & Co KG)","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[2],"policyUrl":"https://www.otto.de/shoppages/service/datenschutz"},{"id":548,"name":"LBC France","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.leboncoin.fr/dc/cookies","deletedDate":"2020-04-23T00:00:00Z"},{"id":569,"name":"Kairos Fire","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"https://www.kairosfire.com/privacy"},{"id":577,"name":"Neustar on behalf of The Procter & Gamble Company","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.pg.com/privacy/english/privacy_statement.shtml"},{"id":590,"name":"Sourcepoint Technologies, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.sourcepoint.com/privacy-policy"},{"id":587,"name":"Localsensor B.V.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[],"policyUrl":"https://www.localsensor.com/privacy.html"},{"id":578,"name":"MAIRDUMONT NETLETIX GmbH&Co. KG","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://mairdumont-netletix.com/datenschutz"},{"id":580,"name":"Goldbach Group AG","purposeIds":[],"legIntPurposeIds":[1,2,3,5],"featureIds":[1,2,3],"policyUrl":"https://goldbach.com/ch/de/datenschutz"},{"id":593,"name":"Programatica de publicidad S.L.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://datmean.com/politica-privacidad/"},{"id":574,"name":"Realeyes OU","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://realview.realeyesit.com/privacy"},{"id":581,"name":"Mobilewalla, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[1,2,3],"policyUrl":"https://www.mobilewalla.com/business-services-privacy-policy"},{"id":598,"name":"audio content & control GmbH","purposeIds":[1],"legIntPurposeIds":[3,4,5],"featureIds":[3],"policyUrl":"https://www.audio-cc.com/audiocc_privacy_policy.pdf"},{"id":596,"name":"InsurAds Technologies SA.","purposeIds":[3],"legIntPurposeIds":[5],"featureIds":[3],"policyUrl":"https://www.insurads.com/privacy.html"},{"id":576,"name":"StartApp Inc.","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[3],"policyUrl":"https://www.startapp.com/policy/privacy-policy/","deletedDate":"2020-04-23T00:00:00Z"},{"id":592,"name":"Colpirio.com","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://privacy-policy.colpirio.com/en/","deletedDate":"2020-03-18T00:00:00Z"},{"id":549,"name":"Bandsintown Amplified LLC","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://corp.bandsintown.com/privacy"},{"id":597,"name":"Better Banners A/S","purposeIds":[],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://betterbanners.com/en/privacy"},{"id":601,"name":"WebAds B.V","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://privacy.webads.eu/"},{"id":599,"name":"Maximus Live LLC","purposeIds":[],"legIntPurposeIds":[3,4,5],"featureIds":[3],"policyUrl":"https://maximusx.com/privacy-policy/"},{"id":604,"name":"Join","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.teamjoin.fr/privacy.html","deletedDate":"2020-04-23T00:00:00Z"},{"id":606,"name":"Impactify ","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[],"policyUrl":"https://impactify.io/privacy-policy/"},{"id":608,"name":"News and Media Holding, a.s.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.newsandmedia.sk/gdpr/"},{"id":602,"name":"Online Solution Int Limited","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2],"policyUrl":"https://adsafety.net/privacy.html"},{"id":591,"name":"Consumable, Inc.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://consumable.com/privacy-policy.html"},{"id":614,"name":"Market Resource Partners LLC","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.mrpfd.com/privacy-policy/"},{"id":615,"name":"Adsolutions BV","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.adsolutions.com/privacy-policy/"},{"id":607,"name":"ucfunnel Co., Ltd.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.ucfunnel.com/privacy-policy"},{"id":609,"name":"Predicio","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.predic.io/privacy"},{"id":617,"name":"Onfocus (Adagio)","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://adagio.io/privacy"},{"id":620,"name":"Blue","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[2,3],"policyUrl":"http://www.getblue.io/privacy/"},{"id":610,"name":"Azerion Holding B.V.","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[2,3],"policyUrl":"https://azerion.com/business/privacy.html"},{"id":621,"name":"Seznam.cz, a.s.","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[2,3],"policyUrl":"https://www.seznam.cz/ochranaudaju"},{"id":624,"name":"Norstat AS","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.norstatpanel.com/en/data-protection"},{"id":623,"name":"Adprime Media Inc. ","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://adprimehealth.com/privacy/","deletedDate":"2020-06-17T00:00:00Z"},{"id":95,"name":"Lotame Solutions, inc","purposeIds":[],"legIntPurposeIds":[1,2],"featureIds":[2],"policyUrl":"https://www.lotame.com/about-lotame/privacy/lotame-corporate-websites-privacy-policy/"},{"id":618,"name":"BEINTOO SPA","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://www.beintoo.com/privacy-cookie-policy/"},{"id":619,"name":"Capitaldata","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.capitaldata.fr/privacy"},{"id":625,"name":"BILENDI SA","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.maximiles.com/privacy-policy"},{"id":628,"name":": Tappx","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.tappx.com/en/privacy-policy/"},{"id":626,"name":"Hivestack Inc.","purposeIds":[],"legIntPurposeIds":[1],"featureIds":[3],"policyUrl":"https://hivestack.com/privacy-policy"},{"id":631,"name":"Relay42 Netherlands B.V.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://relay42.com/privacy"},{"id":627,"name":"D-Edge","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.d-edge.com/privacy-policy/","deletedDate":"2020-07-06T00:00:00Z"},{"id":644,"name":"Gamoshi LTD","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.gamoshi.com/privacy-policy"},{"id":639,"name":"Smile Wanted Group","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.smilewanted.com/privacy.php"},{"id":635,"name":"WebMediaRM","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.webmediarm.com/vie_privee_et_opposition_en.php"},{"id":579,"name":"Ve Global","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.ve.com/privacy-policy"},{"id":645,"name":"Noster Finance S.L.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.finect.com/terminos-legales/politica-de-cookies"},{"id":653,"name":"Smartme Analytics","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[1],"policyUrl":"http://smartmeapp.com/info/smartme/aviso_legal.php","deletedDate":"2020-07-03T00:00:00Z"},{"id":613,"name":"Adserve.zone / Artworx AS","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://adserve.zone/adserveprivacypolicy.html"},{"id":573,"name":"Dailymotion SA","purposeIds":[2,3,4,5],"legIntPurposeIds":[1],"featureIds":[2],"policyUrl":"https://www.dailymotion.com/legal/privacy"},{"id":652,"name":"Skaze","purposeIds":[1,2],"legIntPurposeIds":[3,4,5],"featureIds":[1,2,3],"policyUrl":"http://www.skaze.fr/rgpd/"},{"id":646,"name":"Notify","purposeIds":[1,2],"legIntPurposeIds":[5],"featureIds":[1],"policyUrl":"https://notify-group.com/en/mentions-legales/"},{"id":648,"name":"TrueData Solutions, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.truedata.co/privacy-policy/"},{"id":647,"name":"Axel Springer Teaser Ad GmbH","purposeIds":[2],"legIntPurposeIds":[1,3,5],"featureIds":[],"policyUrl":"https://www.adup-tech.com/privacy"},{"id":654,"name":"GRAPHINIUM","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.graphinium.com/privacy/"},{"id":659,"name":"Research and Analysis of Media in Sweden AB","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www2.rampanel.com/privacy-policy/"},{"id":656,"name":"Think Clever Media","purposeIds":[1,2,3,4],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.contentignite.com/privacy-policy/"},{"id":504,"name":"Alive & Kicking Global Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.mcsaatchiplc.com/legal/privacy-cookies","deletedDate":"2020-07-27T00:00:00Z"},{"id":657,"name":"GP One GmbH","purposeIds":[],"legIntPurposeIds":[1,3,5],"featureIds":[3],"policyUrl":"https://www.gsi-one.org/de/privacy-policy.html"},{"id":655,"name":"Sportradar AG","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.sportradar.com/about-us/privacy/"},{"id":662,"name":"SoundCast","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://soundcast.fm/en/data-privacy"},{"id":665,"name":"Digital East GmbH","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.digitaleast.mobi/en/legal/privacy-policy/"},{"id":650,"name":"Telefonica Investigaci\u00f3n y Desarrollo S.A.U","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://www.cognitivemarketing.tid.es/"},{"id":666,"name":"BeOp","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"https://beop.io/privacy"},{"id":663,"name":"Mobsuccess","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://www.mobsuccess.com/en/privacy"},{"id":658,"name":"BLIINK SAS","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://bliink.io/privacy-policy"},{"id":667,"name":"Liftoff Mobile, Inc.","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[3],"policyUrl":"https://liftoff.io/privacy-policy/"},{"id":668,"name":"WhatRocks Inc. ","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.whatrocks.co/en/privacy-policy "},{"id":670,"name":"Timehop, Inc.","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.timehop.com/privacy"},{"id":674,"name":"Duration Media, LLC.","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.durationmedia.net/privacy-policy"},{"id":675,"name":"Instreamatic inc.","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://instreamatic.com/privacy-policy/"},{"id":676,"name":"BusinessClick","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.businessclick.com/documents/RegulaminProgramuBusinessClick-2019.pdf"},{"id":677,"name":"Intercept Interactive Inc. dba Undertone","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.undertone.com/privacy/"},{"id":660,"name":"Schibsted Norge AS","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[2,3],"policyUrl":"https://static.vg.no/privacy/","deletedDate":"2019-09-16T00:00:00Z"},{"id":673,"name":"TTNET AS","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"http://www.programattik.com/en/privacy-policy.aspx"},{"id":664,"name":"adMarketplace, Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"https://www.admarketplace.com/privacy-policy/"},{"id":671,"name":"Mediaforce LTD","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,3],"policyUrl":"http://casino.mindthebet.co.uk/themes/mindthebetv2-casino/privacy.php"},{"id":561,"name":"AuDigent","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://audigent.com/platform-privacy-policy"},{"id":682,"name":"Radio Net Media Limited","purposeIds":[1,2,3],"legIntPurposeIds":[5],"featureIds":[1,2,3],"policyUrl":"https://www.adtonos.com/service-privacy-policy/"},{"id":684,"name":"Blue Billywig BV","purposeIds":[],"legIntPurposeIds":[5],"featureIds":[],"policyUrl":"https://www.bluebillywig.com/privacy-statement/"},{"id":686,"name":"The MediaGrid Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://www.themediagrid.com/privacy-policy/"},{"id":685,"name":"Arkeero","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://arkeero.com/privacy-2/"},{"id":687,"name":"MISSENA","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://missena.com/confidentialite/"},{"id":690,"name":"Go.pl sp. z o.o.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://go.pl/polityka-prywatnosci/"},{"id":691,"name":"Lifesight Pte. Ltd.","purposeIds":[1,2,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.lifesight.io/privacy-policy/"},{"id":697,"name":"ADWAYS SAS","purposeIds":[],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://www.adways.com/confidentialite/?lang=en"},{"id":681,"name":"MyTraffic","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://www.mytraffic.io/en/privacy"},{"id":649,"name":"adality GmbH","purposeIds":[],"legIntPurposeIds":[1,2],"featureIds":[1],"policyUrl":"https://adality.de/en/privacy/"},{"id":712,"name":"Inspired Mobile Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://byinspired.com/privacypolicy.pdf"},{"id":688,"name":"Effinity","purposeIds":[],"legIntPurposeIds":[1],"featureIds":[],"policyUrl":"https://www.effiliation.com/politique-de-confidentialite/"},{"id":702,"name":"Kwanko","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.kwanko.com/fr/rgpd/"},{"id":715,"name":"BidBerry SRL","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://www.bidberrymedia.com/privacy-policy/"},{"id":713,"name":"Dataseat Ltd","purposeIds":[2,5],"legIntPurposeIds":[1,3,4],"featureIds":[],"policyUrl":"https://dataseat.com/privacy-policy"},{"id":716,"name":"OnAudience Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.onaudience.com/internet-advertising-privacy-policy"},{"id":708,"name":"Dugout Limited ","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://dugout.com/privacy-policy"},{"id":717,"name":"Audience Network","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://www.en.audiencenetwork.pl/internet-advertising-privacy-policy"},{"id":718,"name":"AppConsent Xchange","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://appconsent.io/en/privacy-policy"},{"id":720,"name":"AAX LLC","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[3],"policyUrl":"https://aax.media/privacy/"},{"id":678,"name":"Axonix LTD","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://axonix.com/privacy-cookie-policy/"},{"id":719,"name":"Online Advertising Network Sp. z o.o.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://www.oan.pl/en/privacy-policy"},{"id":707,"name":"Dentsu Aegis Network Italia SpA","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.dentsuaegisnetwork.com/it/it/policies/info-cookie"},{"id":721,"name":"Beaconspark Ltd","purposeIds":[1,2,3],"legIntPurposeIds":[4,5],"featureIds":[1],"policyUrl":"https://www.engageya.com/privacy"},{"id":724,"name":"Between Exchange","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[2,3],"policyUrl":"https://en.betweenx.com/pdata.pdf"},{"id":728,"name":"Appier PTE Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.appier.com/privacy-policy/"},{"id":729,"name":"Cavai AS & UK ","purposeIds":[],"legIntPurposeIds":[3],"featureIds":[],"policyUrl":"https://cav.ai/privacy-policy/"},{"id":723,"name":"Adzymic Pte Ltd","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"http://www.adzymic.co/privacy"},{"id":737,"name":"Monet Engine Inc","purposeIds":[1,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://appmonet.com/privacy-policy/"},{"id":740,"name":"6Sense Insights, Inc.","purposeIds":[1],"legIntPurposeIds":[2,3,4,5],"featureIds":[1,2],"policyUrl":"https://6sense.com/privacy-policy/"},{"id":744,"name":"Vidazoo Ltd","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[2],"policyUrl":"https://vidazoo.gitbook.io/vidazoo-legal/privacy-policy"},{"id":731,"name":"GeistM Technologies LTD","purposeIds":[],"legIntPurposeIds":[3,4,5],"featureIds":[],"policyUrl":"https://www.geistm.com/privacy"},{"id":741,"name":"Brand Advance Limited","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.wearebrandadvance.com/website-privacy-policy"},{"id":734,"name":"Cint AB","purposeIds":[1,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://www.cint.com/participant-privacy-notice"},{"id":709,"name":"NC Audience Exchange, LLC (NewsIQ)","purposeIds":[1,2],"legIntPurposeIds":[3,5],"featureIds":[1,2],"policyUrl":"https://www.ncaudienceexchange.com/privacy/"},{"id":739,"name":"Blingby LLC","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://blingby.com/privacy"},{"id":732,"name":"Performax.cz, s.r.o.","purposeIds":[2,4,5],"legIntPurposeIds":[1,3],"featureIds":[2,3],"policyUrl":"https://reg.tiscali.cz/privacy-policy"},{"id":736,"name":"BidMachine Inc.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://explorestack.com/privacy-policy/"},{"id":738,"name":"adbility media GmbH","purposeIds":[2,3],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.adbility-media.com/datenschutzerklaerung/"},{"id":742,"name":"Audiencerate LTD","purposeIds":[],"legIntPurposeIds":[1,2,5],"featureIds":[],"policyUrl":"https://www.audiencerate.com/privacy/"},{"id":743,"name":"MOVIads Sp. z o.o. Sp. k.","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://moviads.pl/polityka-prywatnosci/"},{"id":746,"name":"Adxperience SAS","purposeIds":[2],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://adxperience.com/privacy-policy/"},{"id":747,"name":"Kairion GmbH","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://kairion.de/datenschutzbestimmungen/"},{"id":748,"name":"AUDIOMOB LTD","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[2,3],"policyUrl":"https://www.audiomob.io/privacy"},{"id":749,"name":"Good-Loop Ltd","purposeIds":[2],"legIntPurposeIds":[1,3,4,5],"featureIds":[],"policyUrl":"https://doc.good-loop.com/policy/privacy-policy.html"},{"id":754,"name":"DistroScale, Inc.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"http://www.distroscale.com/privacy-policy/"},{"id":756,"name":"Fandom, Inc.","purposeIds":[3],"legIntPurposeIds":[1,2,4,5],"featureIds":[],"policyUrl":"https://www.fandom.com/privacy-policy"},{"id":758,"name":"GfK Netherlands B.V.","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://gfkpanel.nl/privacy"},{"id":759,"name":"RevJet","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.revjet.com/privacy"},{"id":760,"name":"VEXPRO TECHNOLOGIES LTD","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2],"policyUrl":"https://onedash.com/privacy-policy.html"},{"id":761,"name":"Digiseg ApS","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[1],"policyUrl":"https://digiseg.io/privacy-center/"},{"id":763,"name":"Delidatax SL","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"https://www.delidatax.net/privacy.htm"},{"id":764,"name":"Lucidity","purposeIds":[1,3,4,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://golucidity.com/privacy-policy/"},{"id":765,"name":"Grabit Interactive Media Inc dba KERV Interctive","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[2],"policyUrl":"https://kervit.com/privacy-policy/"},{"id":766,"name":"ADCELL | Firstlead GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.adcell.de/agb#sector_6"},{"id":768,"name":"Global Media & Entertainment Limited","purposeIds":[1,2,3,4,5],"legIntPurposeIds":[],"featureIds":[1,2,3],"policyUrl":"http://global.com/privacy-policy/"},{"id":770,"name":"MARKETPERF CORP","purposeIds":[1,2,4],"legIntPurposeIds":[3,5],"featureIds":[2,3],"policyUrl":"https://www.marketperf.com/assets/images/app/marketperf/pdf/privacy-policy.pdf"},{"id":773,"name":"360e-com Sp. z o.o.","purposeIds":[1,2,3,5],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.clickonometrics.com/optout/"},{"id":775,"name":"SelectMedia International LTD","purposeIds":[1],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.selectmedia.asia/terms-and-privacy/"},{"id":778,"name":"Discover-Tech ltd","purposeIds":[2,5],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://discover-tech.io/dsp-privacy-policy/"},{"id":779,"name":"Adtarget Medya A.S.","purposeIds":[1,3],"legIntPurposeIds":[],"featureIds":[3],"policyUrl":"https://adtarget.com.tr/adtarget-privacy-policy-2020.pdf"},{"id":780,"name":"Aniview LTD","purposeIds":[1,2,3],"legIntPurposeIds":[],"featureIds":[],"policyUrl":"https://www.aniview.com/privacy-policy/"},{"id":781,"name":"FeedAd GmbH","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[2,3],"policyUrl":"https://feedad.com/privacy/"},{"id":784,"name":"Nubo LTD","purposeIds":[],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"https://www.recod3.com/privacypolicy.php"},{"id":786,"name":"TargetVideo GmbH","purposeIds":[],"legIntPurposeIds":[1,2,3,4,5],"featureIds":[],"policyUrl":"https://www.target-video.com/datenschutz/"},{"id":798,"name":"Adverticum cPlc.","purposeIds":[2,3,4],"legIntPurposeIds":[1,5],"featureIds":[],"policyUrl":"https://adverticum.net/english/privacy-and-data-processing-information/"},{"id":803,"name":"Click Tech Limited","purposeIds":[1],"legIntPurposeIds":[2,3],"featureIds":[1],"policyUrl":"https://en.yeahmobi.com/html/privacypolicy/"},{"id":808,"name":"Pure Local Media GmbH","purposeIds":[],"legIntPurposeIds":[3,5],"featureIds":[],"policyUrl":"https://purelocalmedia.de/?page_id=593"}]} \ No newline at end of file diff --git a/stored_requests/backends/db_fetcher/fetcher.go b/stored_requests/backends/db_fetcher/fetcher.go index 33009e2dc73..c3b71a3be67 100644 --- a/stored_requests/backends/db_fetcher/fetcher.go +++ b/stored_requests/backends/db_fetcher/fetcher.go @@ -93,6 +93,10 @@ func (fetcher *dbFetcher) FetchRequests(ctx context.Context, requestIDs []string return storedRequestData, storedImpData, errs } +func (fetcher *dbFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + return nil, []error{stored_requests.NotFoundError{accountID, "Account"}} +} + func (fetcher *dbFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { return "", nil } diff --git a/stored_requests/backends/empty_fetcher/fetcher.go b/stored_requests/backends/empty_fetcher/fetcher.go index 48ef468abca..6edf3cc4d00 100644 --- a/stored_requests/backends/empty_fetcher/fetcher.go +++ b/stored_requests/backends/empty_fetcher/fetcher.go @@ -3,6 +3,7 @@ package empty_fetcher import ( "context" "encoding/json" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" ) @@ -27,6 +28,10 @@ func (fetcher EmptyFetcher) FetchRequests(ctx context.Context, requestIDs []stri return } +func (fetcher EmptyFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + return nil, []error{stored_requests.NotFoundError{accountID, "Account"}} +} + func (fetcher EmptyFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { return "", nil } diff --git a/stored_requests/backends/file_fetcher/fetcher.go b/stored_requests/backends/file_fetcher/fetcher.go index 6d7cb9caf77..bff94b21e79 100644 --- a/stored_requests/backends/file_fetcher/fetcher.go +++ b/stored_requests/backends/file_fetcher/fetcher.go @@ -33,6 +33,21 @@ func (fetcher *eagerFetcher) FetchRequests(ctx context.Context, requestIDs []str return storedRequests, storedImpressions, errs } +// FetchAccount fetches the host account configuration for a publisher +func (fetcher *eagerFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + if len(accountID) == 0 { + return nil, []error{fmt.Errorf("Cannot look up an empty accountID")} + } + accountJSON, ok := fetcher.FileSystem.Directories["accounts"].Files[accountID] + if !ok { + return nil, []error{stored_requests.NotFoundError{ + ID: accountID, + DataType: "Account", + }} + } + return accountJSON, nil +} + func (fetcher *eagerFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { fileName := primaryAdServer diff --git a/stored_requests/backends/file_fetcher/fetcher_test.go b/stored_requests/backends/file_fetcher/fetcher_test.go index 76f5e494a64..f0900002c8c 100644 --- a/stored_requests/backends/file_fetcher/fetcher_test.go +++ b/stored_requests/backends/file_fetcher/fetcher_test.go @@ -24,6 +24,20 @@ func TestFileFetcher(t *testing.T) { validateImp(t, storedImps) } +func TestAccountFetcher(t *testing.T) { + fetcher, err := NewFileFetcher("./test") + assert.NoError(t, err, "Failed to create test fetcher") + + account, errs := fetcher.FetchAccount(context.Background(), "valid") + assertErrorCount(t, 0, errs) + assert.JSONEq(t, `{"disabled":false, "id":"valid"}`, string(account)) + + account, errs = fetcher.FetchAccount(context.Background(), "nonexistent") + assertErrorCount(t, 1, errs) + assert.Error(t, errs[0]) + assert.Equal(t, stored_requests.NotFoundError{"nonexistent", "Account"}, errs[0]) +} + func TestInvalidDirectory(t *testing.T) { _, err := NewFileFetcher("./nonexistant-directory") if err == nil { diff --git a/stored_requests/backends/file_fetcher/test/accounts/valid.json b/stored_requests/backends/file_fetcher/test/accounts/valid.json new file mode 100644 index 00000000000..2c8bd12af3c --- /dev/null +++ b/stored_requests/backends/file_fetcher/test/accounts/valid.json @@ -0,0 +1,4 @@ +{ + "id": "valid", + "disabled": false +} diff --git a/stored_requests/backends/http_fetcher/fetcher.go b/stored_requests/backends/http_fetcher/fetcher.go index efd85a001e0..5a7d8fa2878 100644 --- a/stored_requests/backends/http_fetcher/fetcher.go +++ b/stored_requests/backends/http_fetcher/fetcher.go @@ -7,6 +7,7 @@ import ( "fmt" "io/ioutil" "net/http" + "net/url" "strings" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" @@ -19,9 +20,13 @@ import ( // // This file expects the endpoint to satisfy the following API: // +// Stored requests // GET {endpoint}?request-ids=["req1","req2"]&imp-ids=["imp1","imp2","imp3"] // -// This endpoint should return a payload like: +// Accounts +// GET {endpoint}?account-ids=["acc1","acc2"] +// +// The above endpoints should return a payload like: // // { // "requests": { @@ -34,12 +39,25 @@ import ( // "imp3": null // If imp3 is not found // } // } +// or +// { +// "accounts": { +// "acc1": { ... config data for acc1 ... }, +// "acc2": { ... config data for acc2 ... }, +// }, +// } // // func NewFetcher(client *http.Client, endpoint string) *HttpFetcher { // Do some work up-front to figure out if the (configurable) endpoint has a query string or not. // When we build requests, we'll either want to add `?request-ids=...&imp-ids=...` _or_ - // `&request-ids=...&imp-ids=...`, depending. + // `&request-ids=...&imp-ids=...`. + + if _, err := url.Parse(endpoint); err != nil { + glog.Fatalf(`Invalid endpoint "%s": %v`, endpoint, err) + } + glog.Infof("Making http_fetcher for endpoint %v", endpoint) + urlPrefix := endpoint if strings.Contains(endpoint, "?") { urlPrefix = urlPrefix + "&" @@ -47,8 +65,6 @@ func NewFetcher(client *http.Client, endpoint string) *HttpFetcher { urlPrefix = urlPrefix + "?" } - glog.Info("Making http_fetcher which calls GET " + urlPrefix + "request-ids=%REQUEST_ID_LIST%&imp-ids=%IMP_ID_LIST%") - return &HttpFetcher{ client: client, Endpoint: urlPrefix, @@ -81,6 +97,70 @@ func (fetcher *HttpFetcher) FetchRequests(ctx context.Context, requestIDs []stri return } +// FetchAccounts retrieves account configurations +// +// Request format is similar to the one for requests: +// GET {endpoint}?account-ids=["account1","account2",...] +// +// The endpoint is expected to respond with a JSON map with accountID -> json.RawMessage +// { +// "account1": { ... account json ... } +// } +// The JSON contents of account config is returned as-is (NOT validated) +func (fetcher *HttpFetcher) FetchAccounts(ctx context.Context, accountIDs []string) (map[string]json.RawMessage, []error) { + if len(accountIDs) == 0 { + return nil, nil + } + httpReq, err := http.NewRequestWithContext(ctx, "GET", fetcher.Endpoint+"account-ids=[\""+strings.Join(accountIDs, "\",\"")+"\"]", nil) + if err != nil { + return nil, []error{ + fmt.Errorf(`Error fetching accounts %v via http: build request failed with %v`, accountIDs, err), + } + } + httpResp, err := ctxhttp.Do(ctx, fetcher.client, httpReq) + if err != nil { + return nil, []error{ + fmt.Errorf(`Error fetching accounts %v via http: %v`, accountIDs, err), + } + } + defer httpResp.Body.Close() + respBytes, err := ioutil.ReadAll(httpResp.Body) + if err != nil { + return nil, []error{ + fmt.Errorf(`Error fetching accounts %v via http: error reading response: %v`, accountIDs, err), + } + } + if httpResp.StatusCode != http.StatusOK { + return nil, []error{ + fmt.Errorf(`Error fetching accounts %v via http: unexpected response status %d`, accountIDs, httpResp.StatusCode), + } + } + var responseData accountsResponseContract + if err = json.Unmarshal(respBytes, &responseData); err != nil { + return nil, []error{ + fmt.Errorf(`Error fetching accounts %v via http: failed to parse response: %v`, accountIDs, err), + } + } + errs := convertNullsToErrs(responseData.Accounts, "Account", []error{}) + return responseData.Accounts, errs +} + +// FetchAccount fetchers a single accountID and returns its corresponding json +func (fetcher *HttpFetcher) FetchAccount(ctx context.Context, accountID string) (accountJSON json.RawMessage, errs []error) { + accountData, errs := fetcher.FetchAccounts(ctx, []string{accountID}) + if len(errs) > 0 { + return nil, errs + } + accountJSON, ok := accountData[accountID] + if !ok { + return nil, []error{stored_requests.NotFoundError{ + ID: accountID, + DataType: "Account", + }} + } + return accountJSON, nil +} + func (fetcher *HttpFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { if fetcher.Categories == nil { fetcher.Categories = make(map[string]map[string]stored_requests.Category) @@ -186,3 +266,7 @@ type responseContract struct { Requests map[string]json.RawMessage `json:"requests"` Imps map[string]json.RawMessage `json:"imps"` } + +type accountsResponseContract struct { + Accounts map[string]json.RawMessage `json:"accounts"` +} diff --git a/stored_requests/backends/http_fetcher/fetcher_test.go b/stored_requests/backends/http_fetcher/fetcher_test.go index dc4076fd4d9..30933181e1d 100644 --- a/stored_requests/backends/http_fetcher/fetcher_test.go +++ b/stored_requests/backends/http_fetcher/fetcher_test.go @@ -9,6 +9,9 @@ import ( "net/http/httptest" "strings" "testing" + "time" + + "github.com/stretchr/testify/assert" ) func TestSingleReq(t *testing.T) { @@ -16,9 +19,9 @@ func TestSingleReq(t *testing.T) { defer close() reqData, impData, errs := fetcher.FetchRequests(context.Background(), []string{"req-1"}, nil) + assert.Empty(t, errs, "Unexpected errors fetching known requests") assertMapKeys(t, reqData, "req-1") - assertMapKeys(t, impData) - assertErrLength(t, errs, 0) + assert.Empty(t, impData, "Unexpected imps returned fetching just requests") } func TestSeveralReqs(t *testing.T) { @@ -26,9 +29,9 @@ func TestSeveralReqs(t *testing.T) { defer close() reqData, impData, errs := fetcher.FetchRequests(context.Background(), []string{"req-1", "req-2"}, nil) + assert.Empty(t, errs, "Unexpected errors fetching known requests") assertMapKeys(t, reqData, "req-1", "req-2") - assertMapKeys(t, impData) - assertErrLength(t, errs, 0) + assert.Empty(t, impData, "Unexpected imps returned fetching just requests") } func TestSingleImp(t *testing.T) { @@ -36,9 +39,9 @@ func TestSingleImp(t *testing.T) { defer close() reqData, impData, errs := fetcher.FetchRequests(context.Background(), nil, []string{"imp-1"}) - assertMapKeys(t, reqData) + assert.Empty(t, errs, "Unexpected errors fetching known imps") + assert.Empty(t, reqData, "Unexpected requests returned fetching just imps") assertMapKeys(t, impData, "imp-1") - assertErrLength(t, errs, 0) } func TestSeveralImps(t *testing.T) { @@ -46,9 +49,9 @@ func TestSeveralImps(t *testing.T) { defer close() reqData, impData, errs := fetcher.FetchRequests(context.Background(), nil, []string{"imp-1", "imp-2"}) - assertMapKeys(t, reqData) + assert.Empty(t, errs, "Unexpected errors fetching known imps") + assert.Empty(t, reqData, "Unexpected requests returned fetching just imps") assertMapKeys(t, impData, "imp-1", "imp-2") - assertErrLength(t, errs, 0) } func TestReqsAndImps(t *testing.T) { @@ -56,9 +59,9 @@ func TestReqsAndImps(t *testing.T) { defer close() reqData, impData, errs := fetcher.FetchRequests(context.Background(), []string{"req-1"}, []string{"imp-1"}) + assert.Empty(t, errs, "Unexpected errors fetching known reqs and imps") assertMapKeys(t, reqData, "req-1") assertMapKeys(t, impData, "imp-1") - assertErrLength(t, errs, 0) } func TestMissingValues(t *testing.T) { @@ -66,9 +69,94 @@ func TestMissingValues(t *testing.T) { defer close() reqData, impData, errs := fetcher.FetchRequests(context.Background(), []string{"req-1", "req-2"}, []string{"imp-1"}) - assertMapKeys(t, reqData) - assertMapKeys(t, impData) - assertErrLength(t, errs, 3) + assert.Empty(t, reqData, "Fetching unknown reqs should return no reqs") + assert.Empty(t, impData, "Fetching unknown imps should return no imps") + assert.Len(t, errs, 3, "Fetching 3 unknown reqs+imps should return 3 errors") +} + +func TestFetchAccounts(t *testing.T) { + fetcher, close := newTestAccountFetcher(t, []string{"acc-1", "acc-2"}) + defer close() + + accData, errs := fetcher.FetchAccounts(context.Background(), []string{"acc-1", "acc-2"}) + assert.Empty(t, errs, "Unexpected error fetching known accounts") + assertMapKeys(t, accData, "acc-1", "acc-2") +} + +func TestFetchAccountsNoData(t *testing.T) { + fetcher, close := newFetcherBrokenBackend() + defer close() + + accData, errs := fetcher.FetchAccounts(context.Background(), []string{"req-1"}) + assert.Len(t, errs, 1, "Fetching unknown account should have returned an error") + assert.Nil(t, accData, "Fetching unknown account should return nil account map") +} + +func TestFetchAccountsBadJSON(t *testing.T) { + fetcher, close := newFetcherBadJSON() + defer close() + + accData, errs := fetcher.FetchAccounts(context.Background(), []string{"req-1"}) + assert.Len(t, errs, 1, "Fetching account with broken json should have returned an error") + assert.Nil(t, accData, "Fetching account with broken json should return nil account map") +} + +func TestFetchAccountsNoIDsProvided(t *testing.T) { + fetcher, close := newTestAccountFetcher(t, []string{"acc-1", "acc-2"}) + defer close() + + accData, errs := fetcher.FetchAccounts(nil, []string{}) + assert.Empty(t, errs, "Unexpected error fetching empty account list") + assert.Nil(t, accData, "Fetching empty account list should return nil") +} + +// Force build request failure by not providing a context +func TestFetchAccountsFailedBuildRequest(t *testing.T) { + fetcher, close := newTestAccountFetcher(t, []string{"acc-1", "acc-2"}) + defer close() + + accData, errs := fetcher.FetchAccounts(nil, []string{"acc-1"}) + assert.Len(t, errs, 1, "Fetching accounts without context should result in error ") + assert.Nil(t, accData, "Fetching accounts without context should return nil") +} + +// Force http error via request timeout +func TestFetchAccountsContextTimeout(t *testing.T) { + fetcher, close := newTestAccountFetcher(t, []string{"acc-1", "acc-2"}) + defer close() + + ctx, cancel := context.WithTimeout(context.Background(), time.Duration(0)) + defer cancel() + accData, errs := fetcher.FetchAccounts(ctx, []string{"acc-1"}) + assert.Len(t, errs, 1, "Expected context timeout error") + assert.Nil(t, accData, "Unexpected account data returned instead of timeout") +} + +func TestFetchAccount(t *testing.T) { + fetcher, close := newTestAccountFetcher(t, []string{"acc-1"}) + defer close() + + account, errs := fetcher.FetchAccount(context.Background(), "acc-1") + assert.Empty(t, errs, "Unexpected error fetching existing account") + assert.JSONEq(t, `"acc-1"`, string(account), "Unexpected account data fetching existing account") +} + +func TestFetchAccountNoData(t *testing.T) { + fetcher, close := newFetcherBrokenBackend() + defer close() + + unknownAccount, errs := fetcher.FetchAccount(context.Background(), "unknown-acc") + assert.NotEmpty(t, errs, "Retrieving unknown account should return error") + assert.Nil(t, unknownAccount, "Retrieving unknown account should return nil json.RawMessage") +} + +func TestFetchAccountNoIDProvided(t *testing.T) { + fetcher, close := newTestAccountFetcher(t, nil) + defer close() + + account, errs := fetcher.FetchAccount(context.Background(), "") + assert.Len(t, errs, 1, "Fetching account with empty id should error") + assert.Nil(t, account, "Fetching account with empty id should return nil") } func TestErrResponse(t *testing.T) { @@ -77,7 +165,7 @@ func TestErrResponse(t *testing.T) { reqData, impData, errs := fetcher.FetchRequests(context.Background(), []string{"req-1"}, []string{"imp-1"}) assertMapKeys(t, reqData) assertMapKeys(t, impData) - assertErrLength(t, errs, 1) + assert.Len(t, errs, 1) } func assertSameContents(t *testing.T, expected map[string]json.RawMessage, actual map[string]json.RawMessage) { @@ -124,6 +212,14 @@ func newFetcherBrokenBackend() (fetcher *HttpFetcher, closer func()) { return NewFetcher(server.Client(), server.URL), server.Close } +func newFetcherBadJSON() (fetcher *HttpFetcher, closer func()) { + handler := func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`broken JSON`)) + } + server := httptest.NewServer(http.HandlerFunc(handler)) + return NewFetcher(server.Client(), server.URL), server.Close +} + func newEmptyFetcher(t *testing.T, expectReqIDs []string, expectImpIDs []string) (fetcher *HttpFetcher, closer func()) { handler := newHandler(t, expectReqIDs, expectImpIDs, jsonifyToNull) server := httptest.NewServer(http.HandlerFunc(handler)) @@ -139,12 +235,12 @@ func newTestFetcher(t *testing.T, expectReqIDs []string, expectImpIDs []string) func newHandler(t *testing.T, expectReqIDs []string, expectImpIDs []string, jsonifier func(string) json.RawMessage) func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) { query := r.URL.Query() - assertMatches(t, query.Get("request-ids"), expectReqIDs) - assertMatches(t, query.Get("imp-ids"), expectImpIDs) - gotReqIDs := richSplit(query.Get("request-ids")) gotImpIDs := richSplit(query.Get("imp-ids")) + assertMatches(t, gotReqIDs, expectReqIDs) + assertMatches(t, gotImpIDs, expectImpIDs) + reqIDResponse := make(map[string]json.RawMessage, len(gotReqIDs)) impIDResponse := make(map[string]json.RawMessage, len(gotImpIDs)) @@ -174,10 +270,43 @@ func newHandler(t *testing.T, expectReqIDs []string, expectImpIDs []string, json } } -func assertMatches(t *testing.T, query string, expected []string) { +func newTestAccountFetcher(t *testing.T, expectAccIDs []string) (fetcher *HttpFetcher, closer func()) { + handler := newAccountHandler(t, expectAccIDs, jsonifyID) + server := httptest.NewServer(http.HandlerFunc(handler)) + return NewFetcher(server.Client(), server.URL), server.Close +} + +func newAccountHandler(t *testing.T, expectAccIDs []string, jsonifier func(string) json.RawMessage) func(w http.ResponseWriter, r *http.Request) { + return func(w http.ResponseWriter, r *http.Request) { + query := r.URL.Query() + gotAccIDs := richSplit(query.Get("account-ids")) + + assertMatches(t, gotAccIDs, expectAccIDs) + + accIDResponse := make(map[string]json.RawMessage, len(gotAccIDs)) + + for _, accID := range gotAccIDs { + if accID != "" { + accIDResponse[accID] = jsonifier(accID) + } + } + + respObj := accountsResponseContract{ + Accounts: accIDResponse, + } + + if respBytes, err := json.Marshal(respObj); err != nil { + t.Errorf("failed to marshal responseContract in test: %v", err) + w.WriteHeader(http.StatusInternalServerError) + } else { + w.Write(respBytes) + } + } +} + +func assertMatches(t *testing.T, queryVals []string, expected []string) { t.Helper() - queryVals := richSplit(query) if len(queryVals) == 1 && queryVals[0] == "" { if len(expected) != 0 { t.Errorf("Expected no query vals, but got %v", queryVals) @@ -250,11 +379,3 @@ func assertMapKeys(t *testing.T, m map[string]json.RawMessage, keys ...string) { } } } - -func assertErrLength(t *testing.T, errs []error, expected int) { - t.Helper() - - if len(errs) != expected { - t.Errorf("Expected %d errors. Got: %v", expected, errs) - } -} diff --git a/stored_requests/caches/cachestest/reliable.go b/stored_requests/caches/cachestest/reliable.go index 59e6683f8b0..a0ab07df431 100644 --- a/stored_requests/caches/cachestest/reliable.go +++ b/stored_requests/caches/cachestest/reliable.go @@ -11,8 +11,6 @@ import ( const ( reqCacheKey = "known-req" reqCacheVal = `{"req":true}` - impCacheKey = "known-imp" - impCacheVal = `{"imp":true}` ) // AssertCacheRobustness runs tests which can be used to validate any Cache that is 100% reliable. @@ -20,84 +18,41 @@ const ( // // The cacheSupplier should be a function which returns a new Cache (with no data inside) on every call. // This will be called from separate Goroutines to make sure that different tests don't conflict. -func AssertCacheRobustness(t *testing.T, cacheSupplier func() stored_requests.Cache) { +func AssertCacheRobustness(t *testing.T, cacheSupplier func() stored_requests.CacheJSON) { t.Run("TestCacheMiss", cacheMissTester(cacheSupplier())) t.Run("TestCacheHit", cacheHitTester(cacheSupplier())) - t.Run("TestCacheMixed", cacheMixedTester(cacheSupplier())) - t.Run("TestCacheOverlap", cacheOverlapTester(cacheSupplier())) t.Run("TestCacheSaveInvalidate", cacheSaveInvalidateTester(cacheSupplier())) } -func cacheMissTester(cache stored_requests.Cache) func(*testing.T) { +func cacheMissTester(cache stored_requests.CacheJSON) func(*testing.T) { return func(t *testing.T) { - storedReqs, storedImps := cache.Get(context.Background(), []string{"unknown"}, nil) - assertMapLength(t, 0, storedReqs) - assertMapLength(t, 0, storedImps) + storedData := cache.Get(context.Background(), []string{"unknown"}) + assertMapLength(t, 0, storedData) } } -func cacheHitTester(cache stored_requests.Cache) func(*testing.T) { +func cacheHitTester(cache stored_requests.CacheJSON) func(*testing.T) { return func(t *testing.T) { cache.Save(context.Background(), map[string]json.RawMessage{ reqCacheKey: json.RawMessage(reqCacheVal), - }, map[string]json.RawMessage{ - impCacheKey: json.RawMessage(impCacheVal), }) - reqData, impData := cache.Get(context.Background(), []string{reqCacheKey}, []string{impCacheKey}) - if len(reqData) != 1 { - t.Errorf("The cache should have returned the data.") - } + reqData := cache.Get(context.Background(), []string{reqCacheKey}) assertMapLength(t, 1, reqData) assertHasValue(t, reqData, reqCacheKey, reqCacheVal) - - assertMapLength(t, 1, impData) - assertHasValue(t, impData, impCacheKey, impCacheVal) - } -} - -func cacheMixedTester(cache stored_requests.Cache) func(*testing.T) { - return func(t *testing.T) { - cache.Save(context.Background(), map[string]json.RawMessage{ - reqCacheKey: json.RawMessage(reqCacheVal), - }, nil) - reqData, impData := cache.Get(context.Background(), []string{reqCacheKey, "unknown-req"}, nil) - assertMapLength(t, 1, reqData) - assertHasValue(t, reqData, reqCacheKey, reqCacheVal) - assertMapLength(t, 0, impData) } } -func cacheOverlapTester(cache stored_requests.Cache) func(*testing.T) { - commonKey := "id" +func cacheSaveInvalidateTester(cache stored_requests.CacheJSON) func(*testing.T) { return func(t *testing.T) { cache.Save(context.Background(), map[string]json.RawMessage{ - commonKey: json.RawMessage(reqCacheVal), - }, map[string]json.RawMessage{ - commonKey: json.RawMessage(impCacheVal), - }) - reqData, impData := cache.Get(context.Background(), []string{commonKey}, []string{commonKey}) - assertMapLength(t, 1, reqData) - assertHasValue(t, reqData, commonKey, reqCacheVal) - assertMapLength(t, 1, impData) - assertHasValue(t, impData, commonKey, impCacheVal) - } -} - -func cacheSaveInvalidateTester(cache stored_requests.Cache) func(*testing.T) { - return func(t *testing.T) { - cache.Save(context.Background(), map[string]json.RawMessage{ - reqCacheKey: json.RawMessage(reqCacheVal), - }, map[string]json.RawMessage{ reqCacheKey: json.RawMessage(reqCacheVal), }) - reqData, impData := cache.Get(context.Background(), []string{reqCacheKey}, []string{reqCacheKey}) + reqData := cache.Get(context.Background(), []string{reqCacheKey}) assertMapLength(t, 1, reqData) - assertMapLength(t, 1, impData) - cache.Invalidate(context.Background(), []string{reqCacheKey}, []string{reqCacheKey}) - reqData, impData = cache.Get(context.Background(), []string{reqCacheKey}, []string{reqCacheKey}) + cache.Invalidate(context.Background(), []string{reqCacheKey}) + reqData = cache.Get(context.Background(), []string{reqCacheKey}) assertMapLength(t, 0, reqData) - assertMapLength(t, 0, impData) } } diff --git a/stored_requests/caches/memory/cache.go b/stored_requests/caches/memory/cache.go index 4262ea21021..288e6c26b71 100644 --- a/stored_requests/caches/memory/cache.go +++ b/stored_requests/caches/memory/cache.go @@ -5,7 +5,6 @@ import ( "encoding/json" "sync" - "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/coocood/freecache" "github.com/golang/glog" @@ -17,68 +16,52 @@ import ( // 2. The cache is too large. This will cause the least recently used items to be evicted. // // For no TTL, use ttlSeconds <= 0 -func NewCache(cfg *config.InMemoryCache) stored_requests.Cache { - return &cache{ - requestDataCache: newCacheForWithLimits(cfg.RequestCacheSize, cfg.TTL, "Request"), - impDataCache: newCacheForWithLimits(cfg.ImpCacheSize, cfg.TTL, "Imp"), - } -} - -func newCacheForWithLimits(size int, ttl int, dataType string) mapLike { +func NewCache(size int, ttl int, dataType string) stored_requests.CacheJSON { if ttl > 0 && size <= 0 { - glog.Fatal("No in-memory caches defined with a finite TTL but unbounded size. Config validation should have caught this. Failing fast because something is buggy.") + // a positive ttl indicates "LRU" cache type, while unlimited size indicates an "unbounded" cache type + glog.Fatalf("unbounded in-memory %s cache with TTL not allowed. Config validation should have caught this. Failing fast because something is buggy.", dataType) } if size > 0 { glog.Infof("Using a Stored %s in-memory cache. Max size: %d bytes. TTL: %d seconds.", dataType, size, ttl) - return &pbsLRUCache{ - Cache: freecache.NewCache(size), - ttlSeconds: ttl, + return &cache{ + dataType: dataType, + cache: &pbsLRUCache{ + Cache: freecache.NewCache(size), + ttlSeconds: ttl, + }, } } else { glog.Infof("Using an unbounded Stored %s in-memory cache.", dataType) - return &pbsSyncMap{&sync.Map{}} + return &cache{ + dataType: dataType, + cache: &pbsSyncMap{&sync.Map{}}, + } } } type cache struct { - requestDataCache mapLike - impDataCache mapLike + dataType string + cache mapLike } -func (c *cache) Get(ctx context.Context, requestIDs []string, impIDs []string) (requestData map[string]json.RawMessage, impData map[string]json.RawMessage) { - requestData = doGet(c.requestDataCache, requestIDs) - impData = doGet(c.impDataCache, impIDs) - return -} - -func doGet(cache mapLike, ids []string) (data map[string]json.RawMessage) { +func (c *cache) Get(ctx context.Context, ids []string) (data map[string]json.RawMessage) { data = make(map[string]json.RawMessage, len(ids)) for _, id := range ids { - if val, ok := cache.Get(id); ok { + if val, ok := c.cache.Get(id); ok { data[id] = val } } return } -func (c *cache) Save(ctx context.Context, storedRequests map[string]json.RawMessage, storedImps map[string]json.RawMessage) { - c.doSave(c.requestDataCache, storedRequests) - c.doSave(c.impDataCache, storedImps) -} - -func (c *cache) doSave(cache mapLike, values map[string]json.RawMessage) { - for id, data := range values { - cache.Set(id, data) +func (c *cache) Save(ctx context.Context, data map[string]json.RawMessage) { + for id, data := range data { + c.cache.Set(id, data) } } -func (c *cache) Invalidate(ctx context.Context, requestIDs []string, impIDs []string) { - doInvalidate(c.requestDataCache, requestIDs) - doInvalidate(c.impDataCache, impIDs) -} - -func doInvalidate(cache mapLike, ids []string) { +func (c *cache) Invalidate(ctx context.Context, ids []string) { for _, id := range ids { - cache.Delete(id) + c.cache.Delete(id) } } diff --git a/stored_requests/caches/memory/cache_test.go b/stored_requests/caches/memory/cache_test.go index 673b9a0c8fe..20ec1239cd2 100644 --- a/stored_requests/caches/memory/cache_test.go +++ b/stored_requests/caches/memory/cache_test.go @@ -7,52 +7,34 @@ import ( "strconv" "testing" - "github.com/PubMatic-OpenWrap/prebid-server/config" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/caches/cachestest" ) func TestLRURobustness(t *testing.T) { - cachestest.AssertCacheRobustness(t, func() stored_requests.Cache { - return NewCache(&config.InMemoryCache{ - RequestCacheSize: 256 * 1024, - ImpCacheSize: 256 * 1024, - TTL: -1, - }) + cachestest.AssertCacheRobustness(t, func() stored_requests.CacheJSON { + return NewCache(256*1024, -1, "TestData") }) } func TestUnboundedRobustness(t *testing.T) { - cachestest.AssertCacheRobustness(t, func() stored_requests.Cache { - return NewCache(&config.InMemoryCache{ - RequestCacheSize: 0, - ImpCacheSize: 0, - TTL: -1, - }) + cachestest.AssertCacheRobustness(t, func() stored_requests.CacheJSON { + return NewCache(0, -1, "TestData") }) } func TestRaceLRUConcurrency(t *testing.T) { - cache := NewCache(&config.InMemoryCache{ - RequestCacheSize: 256 * 1024, - ImpCacheSize: 256 * 1024, - TTL: -1, - }) - + cache := NewCache(256*1024, -1, "TestData") doRaceTest(t, cache) } func TestRaceUnboundedConcurrency(t *testing.T) { - cache := NewCache(&config.InMemoryCache{ - RequestCacheSize: 0, - ImpCacheSize: 0, - TTL: -1, - }) + cache := NewCache(0, -1, "TestData") doRaceTest(t, cache) } -func doRaceTest(t *testing.T, cache stored_requests.Cache) { +func doRaceTest(t *testing.T, cache stored_requests.CacheJSON) { done := make(chan struct{}) sets := [][]int{rand.Perm(100), rand.Perm(100), rand.Perm(100)} @@ -70,26 +52,26 @@ func doRaceTest(t *testing.T, cache stored_requests.Cache) { } } -func readLots(cache stored_requests.Cache, done chan<- struct{}, reads []int) { +func readLots(cache stored_requests.CacheJSON, done chan<- struct{}, reads []int) { var s struct{} for _, i := range reads { - cache.Get(context.Background(), sliceForVal(i), sliceForVal(-i)) + cache.Get(context.Background(), sliceForVal(i)) } done <- s } -func writeLots(cache stored_requests.Cache, done chan<- struct{}, writes []int) { +func writeLots(cache stored_requests.CacheJSON, done chan<- struct{}, writes []int) { var s struct{} for _, i := range writes { - cache.Save(context.Background(), mapForVal(i), mapForVal(-i)) + cache.Save(context.Background(), mapForVal(i)) } done <- s } -func invalidateLots(cache stored_requests.Cache, done chan<- struct{}, invalidates []int) { +func invalidateLots(cache stored_requests.CacheJSON, done chan<- struct{}, invalidates []int) { var s struct{} for _, i := range invalidates { - cache.Invalidate(context.Background(), sliceForVal(i), sliceForVal(-i)) + cache.Invalidate(context.Background(), sliceForVal(i)) } done <- s } diff --git a/stored_requests/caches/nil_cache/nil_cache.go b/stored_requests/caches/nil_cache/nil_cache.go index de29156e3c9..d043ae55c96 100644 --- a/stored_requests/caches/nil_cache/nil_cache.go +++ b/stored_requests/caches/nil_cache/nil_cache.go @@ -8,13 +8,14 @@ import ( // NilCache is a no-op cache which does nothing useful. type NilCache struct{} -func (c *NilCache) Get(ctx context.Context, requestIDs []string, impIDs []string) (map[string]json.RawMessage, map[string]json.RawMessage) { - return nil, nil +func (c *NilCache) Get(ctx context.Context, ids []string) map[string]json.RawMessage { + return make(map[string]json.RawMessage) } -func (c *NilCache) Save(ctx context.Context, storedRequests map[string]json.RawMessage, storedImps map[string]json.RawMessage) { + +func (c *NilCache) Save(ctx context.Context, data map[string]json.RawMessage) { return } -func (c *NilCache) Invalidate(ctx context.Context, requestIDs []string, impIDs []string) { +func (c *NilCache) Invalidate(ctx context.Context, ids []string) { return } diff --git a/stored_requests/config/config.go b/stored_requests/config/config.go index 2d979e4cd35..9310b2522a1 100644 --- a/stored_requests/config/config.go +++ b/stored_requests/config/config.go @@ -20,6 +20,7 @@ import ( apiEvents "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events/api" httpEvents "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events/http" postgresEvents "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events/postgres" + "github.com/PubMatic-OpenWrap/prebid-server/util/task" "github.com/golang/glog" "github.com/julienschmidt/httprouter" ) @@ -41,29 +42,30 @@ type dbConnection struct { // // As a side-effect, it will add some endpoints to the router if the config calls for it. // In the future we should look for ways to simplify this so that it's not doing two things. -func CreateStoredRequests(cfg *config.StoredRequestsSlim, metricsEngine pbsmetrics.MetricsEngine, client *http.Client, router *httprouter.Router, dbc *dbConnection) (fetcher stored_requests.AllFetcher, shutdown func()) { +func CreateStoredRequests(cfg *config.StoredRequests, metricsEngine pbsmetrics.MetricsEngine, client *http.Client, router *httprouter.Router, dbc *dbConnection) (fetcher stored_requests.AllFetcher, shutdown func()) { // Create database connection if given options for one if cfg.Postgres.ConnectionInfo.Database != "" { conn := cfg.Postgres.ConnectionInfo.ConnString() if dbc.conn == "" { - glog.Infof("Connecting to Postgres for Stored Requests. DB=%s, host=%s, port=%d, user=%s", + glog.Infof("Connecting to Postgres for Stored %s. DB=%s, host=%s, port=%d, user=%s", + cfg.DataType(), cfg.Postgres.ConnectionInfo.Database, cfg.Postgres.ConnectionInfo.Host, cfg.Postgres.ConnectionInfo.Port, cfg.Postgres.ConnectionInfo.Username) - db := newPostgresDB(cfg.Postgres.ConnectionInfo) + db := newPostgresDB(cfg.DataType(), cfg.Postgres.ConnectionInfo) dbc.conn = conn dbc.db = db } // Error out if config is trying to use multiple database connections for different stored requests (not supported yet) if conn != dbc.conn { - glog.Fatal("Multiple database connection settings found in Stored Requests config, only a single database connection is currently supported.") + glog.Fatal("Multiple database connection settings found in config, only a single database connection is currently supported.") } } - eventProducers := newEventProducers(cfg, client, dbc.db, router) + eventProducers := newEventProducers(cfg, client, dbc.db, metricsEngine, router) fetcher = newFetcher(cfg, client, dbc.db) var shutdown1 func() @@ -105,10 +107,7 @@ func CreateStoredRequests(cfg *config.StoredRequestsSlim, metricsEngine pbsmetri // // As a side-effect, it will add some endpoints to the router if the config calls for it. // In the future we should look for ways to simplify this so that it's not doing two things. -func NewStoredRequests(cfg *config.Configuration, metricsEngine pbsmetrics.MetricsEngine, client *http.Client, router *httprouter.Router) (db *sql.DB, shutdown func(), fetcher stored_requests.Fetcher, ampFetcher stored_requests.Fetcher, categoriesFetcher stored_requests.CategoryFetcher, videoFetcher stored_requests.Fetcher) { - // Build individual slim options from combined config struct - slimAuction, slimAmp := resolvedStoredRequestsConfig(cfg) - +func NewStoredRequests(cfg *config.Configuration, metricsEngine pbsmetrics.MetricsEngine, client *http.Client, router *httprouter.Router) (db *sql.DB, shutdown func(), fetcher stored_requests.Fetcher, ampFetcher stored_requests.Fetcher, accountsFetcher stored_requests.AccountFetcher, categoriesFetcher stored_requests.CategoryFetcher, videoFetcher stored_requests.Fetcher) { // TODO: Switch this to be set in config defaults //if cfg.CategoryMapping.CacheEvents.Enabled && cfg.CategoryMapping.CacheEvents.Endpoint == "" { // cfg.CategoryMapping.CacheEvents.Endpoint = "/storedrequest/categorymapping" @@ -116,10 +115,11 @@ func NewStoredRequests(cfg *config.Configuration, metricsEngine pbsmetrics.Metri var dbc dbConnection - fetcher1, shutdown1 := CreateStoredRequests(&slimAuction, metricsEngine, client, router, &dbc) - fetcher2, shutdown2 := CreateStoredRequests(&slimAmp, metricsEngine, client, router, &dbc) + fetcher1, shutdown1 := CreateStoredRequests(&cfg.StoredRequests, metricsEngine, client, router, &dbc) + fetcher2, shutdown2 := CreateStoredRequests(&cfg.StoredRequestsAMP, metricsEngine, client, router, &dbc) fetcher3, shutdown3 := CreateStoredRequests(&cfg.CategoryMapping, metricsEngine, client, router, &dbc) fetcher4, shutdown4 := CreateStoredRequests(&cfg.StoredVideo, metricsEngine, client, router, &dbc) + fetcher5, shutdown5 := CreateStoredRequests(&cfg.Accounts, metricsEngine, client, router, &dbc) db = dbc.db @@ -127,59 +127,19 @@ func NewStoredRequests(cfg *config.Configuration, metricsEngine pbsmetrics.Metri ampFetcher = fetcher2.(stored_requests.Fetcher) categoriesFetcher = fetcher3.(stored_requests.CategoryFetcher) videoFetcher = fetcher4.(stored_requests.Fetcher) + accountsFetcher = fetcher5.(stored_requests.AccountFetcher) shutdown = func() { shutdown1() shutdown2() shutdown3() shutdown4() + shutdown5() } return } -func resolvedStoredRequestsConfig(cfg *config.Configuration) (auc, amp config.StoredRequestsSlim) { - sr := &cfg.StoredRequests - - // Auction endpoint uses non-Amp fields so can just copy the slin data - auc.Files.Enabled = sr.Files - auc.Files.Path = sr.Path - auc.Postgres.ConnectionInfo = sr.Postgres.ConnectionInfo - auc.Postgres.FetcherQueries.QueryTemplate = sr.Postgres.FetcherQueries.QueryTemplate - auc.Postgres.CacheInitialization.Timeout = sr.Postgres.CacheInitialization.Timeout - auc.Postgres.CacheInitialization.Query = sr.Postgres.CacheInitialization.Query - auc.Postgres.PollUpdates.RefreshRate = sr.Postgres.PollUpdates.RefreshRate - auc.Postgres.PollUpdates.Timeout = sr.Postgres.PollUpdates.Timeout - auc.Postgres.PollUpdates.Query = sr.Postgres.PollUpdates.Query - auc.HTTP.Endpoint = sr.HTTP.Endpoint - auc.InMemoryCache = sr.InMemoryCache - auc.CacheEvents.Enabled = sr.CacheEventsAPI - auc.CacheEvents.Endpoint = "/storedrequests/openrtb2" - auc.HTTPEvents.RefreshRate = sr.HTTPEvents.RefreshRate - auc.HTTPEvents.Timeout = sr.HTTPEvents.Timeout - auc.HTTPEvents.Endpoint = sr.HTTPEvents.Endpoint - - // Amp endpoint uses all the slim data but some fields get replacyed by Amp* version of similar fields - amp.Files.Enabled = sr.Files - amp.Files.Path = sr.Path - amp.Postgres.ConnectionInfo = sr.Postgres.ConnectionInfo - amp.Postgres.FetcherQueries.QueryTemplate = sr.Postgres.FetcherQueries.AmpQueryTemplate - amp.Postgres.CacheInitialization.Timeout = sr.Postgres.CacheInitialization.Timeout - amp.Postgres.CacheInitialization.Query = sr.Postgres.CacheInitialization.AmpQuery - amp.Postgres.PollUpdates.RefreshRate = sr.Postgres.PollUpdates.RefreshRate - amp.Postgres.PollUpdates.Timeout = sr.Postgres.PollUpdates.Timeout - amp.Postgres.PollUpdates.Query = sr.Postgres.PollUpdates.AmpQuery - amp.HTTP.Endpoint = sr.HTTP.AmpEndpoint - amp.InMemoryCache = sr.InMemoryCache - amp.CacheEvents.Enabled = sr.CacheEventsAPI - amp.CacheEvents.Endpoint = "/storedrequests/amp" - amp.HTTPEvents.RefreshRate = sr.HTTPEvents.RefreshRate - amp.HTTPEvents.Timeout = sr.HTTPEvents.Timeout - amp.HTTPEvents.Endpoint = sr.HTTPEvents.AmpEndpoint - - return -} - func addListeners(cache stored_requests.Cache, eventProducers []events.EventProducer) (shutdown func()) { listeners := make([]*events.EventListener, 0, len(eventProducers)) @@ -196,36 +156,41 @@ func addListeners(cache stored_requests.Cache, eventProducers []events.EventProd } } -func newFetcher(cfg *config.StoredRequestsSlim, client *http.Client, db *sql.DB) (fetcher stored_requests.AllFetcher) { +func newFetcher(cfg *config.StoredRequests, client *http.Client, db *sql.DB) (fetcher stored_requests.AllFetcher) { idList := make(stored_requests.MultiFetcher, 0, 3) if cfg.Files.Enabled { - fFetcher := newFilesystem(cfg.Files.Path) + fFetcher := newFilesystem(cfg.DataType(), cfg.Files.Path) idList = append(idList, fFetcher) } if cfg.Postgres.FetcherQueries.QueryTemplate != "" { - glog.Infof("Loading Stored Requests via Postgres.\nQuery: %s", cfg.Postgres.FetcherQueries.QueryTemplate) + glog.Infof("Loading Stored %s data via Postgres.\nQuery: %s", cfg.DataType(), cfg.Postgres.FetcherQueries.QueryTemplate) idList = append(idList, db_fetcher.NewFetcher(db, cfg.Postgres.FetcherQueries.MakeQuery)) } if cfg.HTTP.Endpoint != "" { - glog.Infof("Loading Stored Requests via HTTP. endpoint=%s", cfg.HTTP.Endpoint) + glog.Infof("Loading Stored %s data via HTTP. endpoint=%s", cfg.DataType(), cfg.HTTP.Endpoint) idList = append(idList, http_fetcher.NewFetcher(client, cfg.HTTP.Endpoint)) } - fetcher = consolidate(idList) + fetcher = consolidate(cfg.DataType(), idList) return } -func newCache(cfg *config.StoredRequestsSlim) stored_requests.Cache { - if cfg.InMemoryCache.Type == "none" { - glog.Info("No Stored Request cache configured. The Fetcher backend will be used for all Stored Requests.") - return &nil_cache.NilCache{} +func newCache(cfg *config.StoredRequests) stored_requests.Cache { + cache := stored_requests.Cache{&nil_cache.NilCache{}, &nil_cache.NilCache{}, &nil_cache.NilCache{}} + switch { + case cfg.InMemoryCache.Type == "none": + glog.Warningf("No %s cache configured. The %s Fetcher backend will be used for all data requests", cfg.DataType(), cfg.DataType()) + case cfg.DataType() == config.AccountDataType: + cache.Accounts = memory.NewCache(cfg.InMemoryCache.Size, cfg.InMemoryCache.TTL, "Accounts") + default: + cache.Requests = memory.NewCache(cfg.InMemoryCache.RequestCacheSize, cfg.InMemoryCache.TTL, "Requests") + cache.Imps = memory.NewCache(cfg.InMemoryCache.ImpCacheSize, cfg.InMemoryCache.TTL, "Imps") } - - return memory.NewCache(&cfg.InMemoryCache) + return cache } -func newEventProducers(cfg *config.StoredRequestsSlim, client *http.Client, db *sql.DB, router *httprouter.Router) (eventProducers []events.EventProducer) { +func newEventProducers(cfg *config.StoredRequests, client *http.Client, db *sql.DB, metricsEngine pbsmetrics.MetricsEngine, router *httprouter.Router) (eventProducers []events.EventProducer) { if cfg.CacheEvents.Enabled { eventProducers = append(eventProducers, newEventsAPI(router, cfg.CacheEvents.Endpoint)) } @@ -233,28 +198,24 @@ func newEventProducers(cfg *config.StoredRequestsSlim, client *http.Client, db * eventProducers = append(eventProducers, newHttpEvents(client, cfg.HTTPEvents.TimeoutDuration(), cfg.HTTPEvents.RefreshRateDuration(), cfg.HTTPEvents.Endpoint)) } if cfg.Postgres.CacheInitialization.Query != "" { - // Make sure we don't miss any updates in between the initial fetch and the "update" polling. - updateStartTime := time.Now() - timeout := time.Duration(cfg.Postgres.CacheInitialization.Timeout) * time.Millisecond - ctx, cancel := context.WithTimeout(context.Background(), timeout) - eventProducers = append(eventProducers, postgresEvents.LoadAll(ctx, db, cfg.Postgres.CacheInitialization.Query)) - cancel() - - if cfg.Postgres.PollUpdates.Query != "" { - eventProducers = append(eventProducers, newPostgresPolling(cfg.Postgres.PollUpdates, db, updateStartTime)) + pgEventCfg := postgresEvents.PostgresEventProducerConfig{ + DB: db, + RequestType: cfg.DataType(), + CacheInitQuery: cfg.Postgres.CacheInitialization.Query, + CacheInitTimeout: time.Duration(cfg.Postgres.CacheInitialization.Timeout) * time.Millisecond, + CacheUpdateQuery: cfg.Postgres.PollUpdates.Query, + CacheUpdateTimeout: time.Duration(cfg.Postgres.PollUpdates.Timeout) * time.Millisecond, + MetricsEngine: metricsEngine, } + pgEventProducer := postgresEvents.NewPostgresEventProducer(pgEventCfg) + fetchInterval := time.Duration(cfg.Postgres.PollUpdates.RefreshRate) * time.Second + pgEventTickerTask := task.NewTickerTask(fetchInterval, pgEventProducer) + pgEventTickerTask.Start() + eventProducers = append(eventProducers, pgEventProducer) } return } -func newPostgresPolling(cfg config.PostgresUpdatePollingSlim, db *sql.DB, startTime time.Time) events.EventProducer { - timeout := time.Duration(cfg.Timeout) * time.Millisecond - ctxProducer := func() (ctx context.Context, canceller func()) { - return context.WithTimeout(context.Background(), timeout) - } - return postgresEvents.PollForUpdates(ctxProducer, db, cfg.Query, startTime, time.Duration(cfg.RefreshRate)*time.Second) -} - func newEventsAPI(router *httprouter.Router, endpoint string) events.EventProducer { producer, handler := apiEvents.NewEventsAPI() router.POST(endpoint, handler) @@ -269,32 +230,37 @@ func newHttpEvents(client *http.Client, timeout time.Duration, refreshRate time. return httpEvents.NewHTTPEvents(client, endpoint, ctxProducer, refreshRate) } -func newFilesystem(configPath string) stored_requests.AllFetcher { - glog.Infof("Loading Stored Requests from filesystem at path %s", configPath) +func newFilesystem(dataType config.DataType, configPath string) stored_requests.AllFetcher { + glog.Infof("Loading Stored %s data from filesystem at path %s", dataType, configPath) fetcher, err := file_fetcher.NewFileFetcher(configPath) if err != nil { - glog.Fatalf("Failed to create a FileFetcher: %v", err) + glog.Fatalf("Failed to create a %s FileFetcher: %v", dataType, err) } return fetcher } -func newPostgresDB(cfg config.PostgresConnection) *sql.DB { +func newPostgresDB(dataType config.DataType, cfg config.PostgresConnection) *sql.DB { db, err := sql.Open("postgres", cfg.ConnString()) if err != nil { - glog.Fatalf("Failed to open postgres connection: %v", err) + glog.Fatalf("Failed to open %s postgres connection: %v", dataType, err) } if err := db.Ping(); err != nil { - glog.Fatalf("Failed to ping postgres: %v", err) + glog.Fatalf("Failed to ping %s postgres: %v", dataType, err) } return db } // consolidate returns a single Fetcher from an array of fetchers of any size. -func consolidate(fetchers []stored_requests.AllFetcher) stored_requests.AllFetcher { +func consolidate(dataType config.DataType, fetchers []stored_requests.AllFetcher) stored_requests.AllFetcher { if len(fetchers) == 0 { - glog.Warning("No Stored Request support configured. request.imp[i].ext.prebid.storedrequest will be ignored. If you need this, check your app config") + switch dataType { + case config.RequestDataType: + glog.Warning("No Stored Request support configured. request.imp[i].ext.prebid.storedrequest will be ignored. If you need this, check your app config") + default: + glog.Warningf("No Stored %s support configured. If you need this, check your app config", dataType) + } return empty_fetcher.EmptyFetcher{} } else if len(fetchers) == 1 { return fetchers[0] diff --git a/stored_requests/config/config_test.go b/stored_requests/config/config_test.go index e40c0fea733..5c393bb7047 100644 --- a/stored_requests/config/config_test.go +++ b/stored_requests/config/config_test.go @@ -9,141 +9,60 @@ import ( "regexp" "testing" + "github.com/stretchr/testify/assert" + sqlmock "github.com/DATA-DOG/go-sqlmock" "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/empty_fetcher" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/backends/http_fetcher" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events" httpEvents "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events/http" "github.com/julienschmidt/httprouter" + "github.com/stretchr/testify/mock" ) +func typedConfig(dataType config.DataType, sr *config.StoredRequests) *config.StoredRequests { + sr.SetDataType(dataType) + return sr +} + +func isEmptyCacheType(cache stored_requests.CacheJSON) bool { + cache.Save(context.Background(), map[string]json.RawMessage{"foo": json.RawMessage("true")}) + objs := cache.Get(context.Background(), []string{"foo"}) + return len(objs) == 0 +} + +func isMemoryCacheType(cache stored_requests.CacheJSON) bool { + cache.Save(context.Background(), map[string]json.RawMessage{"foo": json.RawMessage("true")}) + objs := cache.Get(context.Background(), []string{"foo"}) + return len(objs) == 1 +} + func TestNewEmptyFetcher(t *testing.T) { - fetcher := newFetcher(&config.StoredRequestsSlim{}, nil, nil) - ampFetcher := newFetcher(&config.StoredRequestsSlim{}, nil, nil) - if fetcher == nil || ampFetcher == nil { - t.Errorf("The fetchers should be non-nil, even with an empty config.") + fetcher := newFetcher(&config.StoredRequests{}, nil, nil) + if fetcher == nil { + t.Errorf("The fetcher should be non-nil, even with an empty config.") } if _, ok := fetcher.(empty_fetcher.EmptyFetcher); !ok { t.Errorf("If the config is empty, and EmptyFetcher should be returned") } - if _, ok := ampFetcher.(empty_fetcher.EmptyFetcher); !ok { - t.Errorf("If the config is empty, and EmptyFetcher should be returned for AMP") - } } func TestNewHTTPFetcher(t *testing.T) { - fetcher := newFetcher(&config.StoredRequestsSlim{ - HTTP: config.HTTPFetcherConfigSlim{ - Endpoint: "stored-requests.prebid.com", - }, - }, nil, nil) - ampFetcher := newFetcher(&config.StoredRequestsSlim{ - HTTP: config.HTTPFetcherConfigSlim{ - Endpoint: "stored-requests.prebid.com?type=amp", - }, - }, nil, nil) - if httpFetcher, ok := fetcher.(*http_fetcher.HttpFetcher); ok { - if httpFetcher.Endpoint != "stored-requests.prebid.com?" { - t.Errorf("The HTTP fetcher is using the wrong endpoint. Expected %s, got %s", "stored-requests.prebid.com?", httpFetcher.Endpoint) - } - } else { - t.Errorf("An HTTP Fetching config should return an HTTPFetcher. Got %v", ampFetcher) - } - if httpFetcher, ok := ampFetcher.(*http_fetcher.HttpFetcher); ok { - if httpFetcher.Endpoint != "stored-requests.prebid.com?type=amp&" { - t.Errorf("The AMP HTTP fetcher is using the wrong endpoint. Expected %s, got %s", "stored-requests.prebid.com?type=amp&", httpFetcher.Endpoint) - } - } else { - t.Errorf("An HTTP Fetching config should return an HTTPFetcher. Got %v", ampFetcher) - } -} - -func TestNewHTTPFetcherNoAmp(t *testing.T) { - fetcher := newFetcher(&config.StoredRequestsSlim{ - HTTP: config.HTTPFetcherConfigSlim{ + fetcher := newFetcher(&config.StoredRequests{ + HTTP: config.HTTPFetcherConfig{ Endpoint: "stored-requests.prebid.com", }, }, nil, nil) - ampFetcher := newFetcher(&config.StoredRequestsSlim{ - HTTP: config.HTTPFetcherConfigSlim{ - Endpoint: "", - }, - }, nil, nil) if httpFetcher, ok := fetcher.(*http_fetcher.HttpFetcher); ok { if httpFetcher.Endpoint != "stored-requests.prebid.com?" { t.Errorf("The HTTP fetcher is using the wrong endpoint. Expected %s, got %s", "stored-requests.prebid.com?", httpFetcher.Endpoint) } } else { - t.Errorf("An HTTP Fetching config should return an HTTPFetcher. Got %v", ampFetcher) - } - if httpAmpFetcher, ok := ampFetcher.(*http_fetcher.HttpFetcher); ok && httpAmpFetcher == nil { - t.Errorf("An HTTP Fetching config should not return an Amp HTTP fetcher in this case. Got %v (%v)", ampFetcher, httpAmpFetcher) - } -} - -func TestResolveConfig(t *testing.T) { - cfg := &config.Configuration{ - StoredRequests: config.StoredRequests{ - Files: true, - Path: "/test-path", - Postgres: config.PostgresConfig{ - ConnectionInfo: config.PostgresConnection{ - Database: "db", - Host: "pghost", - Port: 5, - Username: "user", - Password: "pass", - }, - FetcherQueries: config.PostgresFetcherQueries{ - AmpQueryTemplate: "amp-fetcher-query", - }, - CacheInitialization: config.PostgresCacheInitializer{ - AmpQuery: "amp-cache-init-query", - }, - PollUpdates: config.PostgresUpdatePolling{ - AmpQuery: "amp-poll-query", - }, - }, - HTTP: config.HTTPFetcherConfig{ - AmpEndpoint: "amp-http-fetcher-endpoint", - }, - InMemoryCache: config.InMemoryCache{ - Type: "none", - TTL: 50, - RequestCacheSize: 1, - ImpCacheSize: 2, - }, - CacheEventsAPI: true, - HTTPEvents: config.HTTPEventsConfig{ - AmpEndpoint: "amp-http-events-endpoint", - }, - }, + t.Errorf("An HTTP Fetching config should return an HTTPFetcher. Got %v", fetcher) } - - cfg.StoredRequests.Postgres.FetcherQueries.QueryTemplate = "auc-fetcher-query" - cfg.StoredRequests.Postgres.CacheInitialization.Query = "auc-cache-init-query" - cfg.StoredRequests.Postgres.PollUpdates.Query = "auc-poll-query" - cfg.StoredRequests.HTTP.Endpoint = "auc-http-fetcher-endpoint" - cfg.StoredRequests.HTTPEvents.Endpoint = "auc-http-events-endpoint" - - auc, amp := resolvedStoredRequestsConfig(cfg) - - // Auction slim should have the non-amp values in it - assertStringsEqual(t, auc.Postgres.FetcherQueries.QueryTemplate, cfg.StoredRequests.Postgres.FetcherQueries.QueryTemplate) - assertStringsEqual(t, auc.Postgres.CacheInitialization.Query, cfg.StoredRequests.Postgres.CacheInitialization.Query) - assertStringsEqual(t, auc.Postgres.PollUpdates.Query, cfg.StoredRequests.Postgres.PollUpdates.Query) - assertStringsEqual(t, auc.HTTP.Endpoint, cfg.StoredRequests.HTTP.Endpoint) - assertStringsEqual(t, auc.HTTPEvents.Endpoint, cfg.StoredRequests.HTTPEvents.Endpoint) - assertStringsEqual(t, auc.CacheEvents.Endpoint, "/storedrequests/openrtb2") - - // Amp slim should have the amp values in it - assertStringsEqual(t, amp.Postgres.FetcherQueries.QueryTemplate, cfg.StoredRequests.Postgres.FetcherQueries.AmpQueryTemplate) - assertStringsEqual(t, amp.Postgres.CacheInitialization.Query, cfg.StoredRequests.Postgres.CacheInitialization.AmpQuery) - assertStringsEqual(t, amp.Postgres.PollUpdates.Query, cfg.StoredRequests.Postgres.PollUpdates.AmpQuery) - assertStringsEqual(t, amp.HTTP.Endpoint, cfg.StoredRequests.HTTP.AmpEndpoint) - assertStringsEqual(t, amp.HTTPEvents.Endpoint, cfg.StoredRequests.HTTPEvents.AmpEndpoint) - assertStringsEqual(t, amp.CacheEvents.Endpoint, "/storedrequests/amp") } func TestNewHTTPEvents(t *testing.T) { @@ -152,84 +71,83 @@ func TestNewHTTPEvents(t *testing.T) { } server1 := httptest.NewServer(http.HandlerFunc(handler)) - cfg := &config.StoredRequestsSlim{ - HTTPEvents: config.HTTPEventsConfigSlim{ + cfg := &config.StoredRequests{ + HTTPEvents: config.HTTPEventsConfig{ Endpoint: server1.URL, RefreshRate: 100, Timeout: 1000, }, } - evProducers := newEventProducers(cfg, server1.Client(), nil, nil) + + metricsMock := &pbsmetrics.MetricsEngineMock{} + + evProducers := newEventProducers(cfg, server1.Client(), nil, metricsMock, nil) assertSliceLength(t, evProducers, 1) assertHttpWithURL(t, evProducers[0], server1.URL) } func TestNewEmptyCache(t *testing.T) { - cache := newCache(&config.StoredRequestsSlim{InMemoryCache: config.InMemoryCache{Type: "none"}}) - cache.Save(context.Background(), map[string]json.RawMessage{"foo": json.RawMessage("true")}, nil) - reqs, _ := cache.Get(context.Background(), []string{"foo"}, nil) - if len(reqs) != 0 { - t.Errorf("The newCache method should return an empty cache if the config asks for it.") - } + cache := newCache(&config.StoredRequests{InMemoryCache: config.InMemoryCache{Type: "none"}}) + assert.True(t, isEmptyCacheType(cache.Requests), "The newCache method should return an empty Request cache") + assert.True(t, isEmptyCacheType(cache.Imps), "The newCache method should return an empty Imp cache") + assert.True(t, isEmptyCacheType(cache.Accounts), "The newCache method should return an empty Account cache") } func TestNewInMemoryCache(t *testing.T) { - cache := newCache(&config.StoredRequestsSlim{ + cache := newCache(&config.StoredRequests{ InMemoryCache: config.InMemoryCache{ TTL: 60, RequestCacheSize: 100, ImpCacheSize: 100, }, }) - cache.Save(context.Background(), map[string]json.RawMessage{"foo": json.RawMessage("true")}, nil) - reqs, _ := cache.Get(context.Background(), []string{"foo"}, nil) - if len(reqs) != 1 { - t.Errorf("The newCache method should return an in-memory cache if the config asks for it.") - } + assert.True(t, isMemoryCacheType(cache.Requests), "The newCache method should return an in-memory Request cache for StoredRequests config") + assert.True(t, isMemoryCacheType(cache.Imps), "The newCache method should return an in-memory Imp cache for StoredRequests config") + assert.True(t, isEmptyCacheType(cache.Accounts), "The newCache method should return an empty Account cache for StoredRequests config") +} + +func TestNewInMemoryAccountCache(t *testing.T) { + cache := newCache(typedConfig(config.AccountDataType, &config.StoredRequests{ + InMemoryCache: config.InMemoryCache{ + TTL: 60, + Size: 100, + }, + })) + assert.True(t, isMemoryCacheType(cache.Accounts), "The newCache method should return an in-memory Account cache for Accounts config") + assert.True(t, isEmptyCacheType(cache.Requests), "The newCache method should return an empty Request cache for Accounts config") + assert.True(t, isEmptyCacheType(cache.Imps), "The newCache method should return an empty Imp cache for Accounts config") } func TestNewPostgresEventProducers(t *testing.T) { - cfg := &config.StoredRequestsSlim{ - Postgres: config.PostgresConfigSlim{ - CacheInitialization: config.PostgresCacheInitializerSlim{ + metricsMock := &pbsmetrics.MetricsEngineMock{} + metricsMock.Mock.On("RecordStoredDataFetchTime", mock.Anything, mock.Anything).Return() + metricsMock.Mock.On("RecordStoredDataError", mock.Anything).Return() + + cfg := &config.StoredRequests{ + Postgres: config.PostgresConfig{ + CacheInitialization: config.PostgresCacheInitializer{ Timeout: 50, Query: "SELECT id, requestData, type FROM stored_data", }, - PollUpdates: config.PostgresUpdatePollingSlim{ + PollUpdates: config.PostgresUpdatePolling{ RefreshRate: 20, Timeout: 50, Query: "SELECT id, requestData, type FROM stored_data WHERE last_updated > $1", }, }, } - ampCfg := &config.StoredRequestsSlim{ - Postgres: config.PostgresConfigSlim{ - CacheInitialization: config.PostgresCacheInitializerSlim{ - Timeout: 50, - Query: "SELECT id, requestData, type FROM stored_amp_data", - }, - PollUpdates: config.PostgresUpdatePollingSlim{ - RefreshRate: 20, - Timeout: 50, - Query: "SELECT id, requestData, type FROM stored_amp_data WHERE last_updated > $1", - }, - }, - } client := &http.Client{} db, mock, err := sqlmock.New() if err != nil { t.Fatalf("Failed to create mock: %v", err) } mock.ExpectQuery("^" + regexp.QuoteMeta(cfg.Postgres.CacheInitialization.Query) + "$").WillReturnError(errors.New("Query failed")) - mock.ExpectQuery("^" + regexp.QuoteMeta(ampCfg.Postgres.CacheInitialization.Query) + "$").WillReturnError(errors.New("Query failed")) - - evProducers := newEventProducers(cfg, client, db, nil) - assertProducerLength(t, evProducers, 2) - ampEvProducers := newEventProducers(ampCfg, client, db, nil) - assertProducerLength(t, ampEvProducers, 2) + evProducers := newEventProducers(cfg, client, db, metricsMock, nil) + assertProducerLength(t, evProducers, 1) assertExpectationsMet(t, mock) + metricsMock.AssertExpectations(t) } func TestNewEventsAPI(t *testing.T) { diff --git a/stored_requests/data/by_id/accounts/test.json b/stored_requests/data/by_id/accounts/test.json new file mode 100644 index 00000000000..76bafff7f1c --- /dev/null +++ b/stored_requests/data/by_id/accounts/test.json @@ -0,0 +1,14 @@ +{ + "id": "test", + "name": "test account", + "disabled": true, + "cache_ttl": { + "banner": 600, + "video": 3600, + "native": 3600, + "audio": 3600 + }, + "events": { + "enabled": true + } +} diff --git a/stored_requests/events/api/api_test.go b/stored_requests/events/api/api_test.go index eee6143de10..74e02e69e4d 100644 --- a/stored_requests/events/api/api_test.go +++ b/stored_requests/events/api/api_test.go @@ -9,22 +9,22 @@ import ( "strings" "testing" - "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/caches/memory" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events" ) func TestGoodRequests(t *testing.T) { - cache := memory.NewCache(&config.InMemoryCache{ - RequestCacheSize: 256 * 1024, - ImpCacheSize: 256 * 1024, - TTL: -1, - }) - + cache := stored_requests.Cache{ + Requests: memory.NewCache(256*1024, -1, "Request"), + Imps: memory.NewCache(256*1024, -1, "Imp"), + Accounts: memory.NewCache(256*1024, -1, "Account"), + } id := "1" config := fmt.Sprintf(`{"id": "%s"}`, id) initialValue := map[string]json.RawMessage{id: json.RawMessage(config)} - cache.Save(context.Background(), initialValue, initialValue) + cache.Requests.Save(context.Background(), initialValue) + cache.Imps.Save(context.Background(), initialValue) apiEvents, endpoint := NewEventsAPI() @@ -51,7 +51,8 @@ func TestGoodRequests(t *testing.T) { } <-updateOccurred - reqData, impData := cache.Get(context.Background(), []string{id}, []string{id}) + reqData := cache.Requests.Get(context.Background(), []string{id}) + impData := cache.Imps.Get(context.Background(), []string{id}) assertHasValue(t, reqData, id, config) assertHasValue(t, impData, id, config) @@ -66,18 +67,17 @@ func TestGoodRequests(t *testing.T) { } <-invalidateOccurred - reqData, impData = cache.Get(context.Background(), []string{id}, []string{id}) + reqData = cache.Requests.Get(context.Background(), []string{id}) + impData = cache.Imps.Get(context.Background(), []string{id}) assertMapLength(t, 0, reqData) assertMapLength(t, 0, impData) } func TestBadRequests(t *testing.T) { - cache := memory.NewCache(&config.InMemoryCache{ - RequestCacheSize: 256 * 1024, - ImpCacheSize: 256 * 1024, - TTL: -1, - }) - + cache := stored_requests.Cache{ + Requests: memory.NewCache(256*1024, -1, "Requests"), + Imps: memory.NewCache(256*1024, -1, "Imps"), + } apiEvents, endpoint := NewEventsAPI() listener := events.SimpleEventListener() go listener.Listen(cache, apiEvents) diff --git a/stored_requests/events/events.go b/stored_requests/events/events.go index 2e8dd07c880..60909a0d426 100644 --- a/stored_requests/events/events.go +++ b/stored_requests/events/events.go @@ -11,12 +11,14 @@ import ( type Save struct { Requests map[string]json.RawMessage `json:"requests"` Imps map[string]json.RawMessage `json:"imps"` + Accounts map[string]json.RawMessage `json:"accounts"` } // Invalidation represents a bulk invalidation type Invalidation struct { Requests []string `json:"requests"` Imps []string `json:"imps"` + Accounts []string `json:"accounts"` } // EventProducer will produce cache update and invalidation events on its channels @@ -61,12 +63,16 @@ func (e *EventListener) Listen(cache stored_requests.Cache, events EventProducer for { select { case save := <-events.Saves(): - cache.Save(context.Background(), save.Requests, save.Imps) + cache.Requests.Save(context.Background(), save.Requests) + cache.Imps.Save(context.Background(), save.Imps) + cache.Accounts.Save(context.Background(), save.Accounts) if e.onSave != nil { e.onSave() } case invalidation := <-events.Invalidations(): - cache.Invalidate(context.Background(), invalidation.Requests, invalidation.Imps) + cache.Requests.Invalidate(context.Background(), invalidation.Requests) + cache.Imps.Invalidate(context.Background(), invalidation.Imps) + cache.Accounts.Invalidate(context.Background(), invalidation.Accounts) if e.onInvalidate != nil { e.onInvalidate() } diff --git a/stored_requests/events/events_test.go b/stored_requests/events/events_test.go index 84bbc1c6b13..0a48b4cc365 100644 --- a/stored_requests/events/events_test.go +++ b/stored_requests/events/events_test.go @@ -7,7 +7,7 @@ import ( "reflect" "testing" - "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests" "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/caches/memory" ) @@ -16,12 +16,11 @@ func TestListen(t *testing.T) { saves: make(chan Save), invalidations: make(chan Invalidation), } - - cache := memory.NewCache(&config.InMemoryCache{ - RequestCacheSize: 256 * 1024, - ImpCacheSize: 256 * 1024, - TTL: -1, - }) + cache := stored_requests.Cache{ + Requests: memory.NewCache(256*1024, -1, "Requests"), + Imps: memory.NewCache(256*1024, -1, "Imps"), + Accounts: memory.NewCache(256*1024, -1, "Account"), + } // create channels to synchronize saveOccurred := make(chan struct{}) @@ -41,34 +40,43 @@ func TestListen(t *testing.T) { save := Save{ Requests: data, Imps: data, + Accounts: data, } - cache.Save(context.Background(), save.Requests, save.Imps) + cache.Requests.Save(context.Background(), save.Requests) + cache.Imps.Save(context.Background(), save.Imps) + cache.Accounts.Save(context.Background(), save.Accounts) config = fmt.Sprintf(`{"id": "%s", "updated": true}`, id) data = map[string]json.RawMessage{id: json.RawMessage(config)} save = Save{ Requests: data, Imps: data, + Accounts: data, } ep.saves <- save <-saveOccurred - requestData, impData := cache.Get(context.Background(), idSlice, idSlice) - if !reflect.DeepEqual(requestData, data) || !reflect.DeepEqual(impData, data) { + requestData := cache.Requests.Get(context.Background(), idSlice) + impData := cache.Imps.Get(context.Background(), idSlice) + accountData := cache.Accounts.Get(context.Background(), idSlice) + if !reflect.DeepEqual(requestData, data) || !reflect.DeepEqual(impData, data) || !reflect.DeepEqual(accountData, data) { t.Error("Update failed") } invalidation := Invalidation{ Requests: idSlice, Imps: idSlice, + Accounts: idSlice, } ep.invalidations <- invalidation <-invalidateOccurred - requestData, impData = cache.Get(context.Background(), idSlice, idSlice) - if len(requestData) > 0 || len(impData) > 0 { + requestData = cache.Requests.Get(context.Background(), idSlice) + impData = cache.Imps.Get(context.Background(), idSlice) + accountData = cache.Accounts.Get(context.Background(), idSlice) + if len(requestData) > 0 || len(impData) > 0 || len(accountData) > 0 { t.Error("Invalidate failed") } } diff --git a/stored_requests/events/http/http.go b/stored_requests/events/http/http.go index a9f26d0c9d2..790c247e368 100644 --- a/stored_requests/events/http/http.go +++ b/stored_requests/events/http/http.go @@ -42,6 +42,13 @@ import ( // "imp2": { ... stored data for imp2 ... }, // } // } +// or +// { +// "accounts": { +// "acc1": { ... config data for acc1 ... }, +// "acc2": { ... config data for acc2 ... }, +// }, +// } // // To signal deletions, the endpoint may return { "deleted": true } // in place of the Stored Data if the "last-modified" param existed. @@ -82,10 +89,11 @@ func (e *HTTPEvents) fetchAll() { defer cancel() resp, err := ctxhttp.Get(ctx, e.client, e.Endpoint) if respObj, ok := e.parse(e.Endpoint, resp, err); ok && - (len(respObj.StoredRequests) > 0 || len(respObj.StoredImps) > 0) { + (len(respObj.StoredRequests) > 0 || len(respObj.StoredImps) > 0 || len(respObj.Accounts) > 0) { e.saves <- events.Save{ Requests: respObj.StoredRequests, Imps: respObj.StoredImps, + Accounts: respObj.Accounts, } } } @@ -125,14 +133,16 @@ func (e *HTTPEvents) refresh(ticker <-chan time.Time) { invalidations := events.Invalidation{ Requests: extractInvalidations(respObj.StoredRequests), Imps: extractInvalidations(respObj.StoredImps), + Accounts: extractInvalidations(respObj.Accounts), } - if len(respObj.StoredRequests) > 0 || len(respObj.StoredImps) > 0 { + if len(respObj.StoredRequests) > 0 || len(respObj.StoredImps) > 0 || len(respObj.Accounts) > 0 { e.saves <- events.Save{ Requests: respObj.StoredRequests, Imps: respObj.StoredImps, + Accounts: respObj.Accounts, } } - if len(invalidations.Requests) > 0 || len(invalidations.Imps) > 0 { + if len(invalidations.Requests) > 0 || len(invalidations.Imps) > 0 || len(invalidations.Accounts) > 0 { e.invalidations <- invalidations } e.lastUpdate = thisTimeInUTC @@ -193,4 +203,5 @@ func (e *HTTPEvents) Invalidations() <-chan events.Invalidation { type responseContract struct { StoredRequests map[string]json.RawMessage `json:"requests"` StoredImps map[string]json.RawMessage `json:"imps"` + Accounts map[string]json.RawMessage `json:"accounts"` } diff --git a/stored_requests/events/http/http_test.go b/stored_requests/events/http/http_test.go index fdba84cd6fe..2a1aa5d8dfc 100644 --- a/stored_requests/events/http/http_test.go +++ b/stored_requests/events/http/http_test.go @@ -1,145 +1,161 @@ package http import ( - "bytes" "context" "encoding/json" + "fmt" httpCore "net/http" "net/http/httptest" "testing" "time" -) - -func TestStartupReqsOnly(t *testing.T) { - server := httptest.NewServer(&mockResponseHandler{ - statusCode: httpCore.StatusOK, - response: `{"requests":{"request1":{"value":1}, "request2":{"value":2}}}`, - }) - defer server.Close() - - ev := NewHTTPEvents(server.Client(), server.URL, nil, -1) - theSave := <-ev.Saves() - - assertLen(t, theSave.Requests, 2) - assertHasValue(t, theSave.Requests, "request1", `{"value":1}`) - assertHasValue(t, theSave.Requests, "request2", `{"value":2}`) - - assertLen(t, theSave.Imps, 0) -} - -func TestStartupImpsOnly(t *testing.T) { - server := httptest.NewServer(&mockResponseHandler{ - statusCode: httpCore.StatusOK, - response: `{"imps":{"imp1":{"value":1}}}`, - }) - defer server.Close() - - ev := NewHTTPEvents(server.Client(), server.URL, nil, -1) - theSave := <-ev.Saves() - - assertLen(t, theSave.Requests, 0) - - assertLen(t, theSave.Imps, 1) - assertHasValue(t, theSave.Imps, "imp1", `{"value":1}`) -} - -func TestStartupBothTypes(t *testing.T) { - server := httptest.NewServer(&mockResponseHandler{ - statusCode: httpCore.StatusOK, - response: `{"requests":{"request1":{"value":1}, "request2":{"value":2}},"imps":{"imp1":{"value":1}}}`, - }) - defer server.Close() - - ev := NewHTTPEvents(server.Client(), server.URL, nil, -1) - theSave := <-ev.Saves() - - assertLen(t, theSave.Requests, 2) - assertHasValue(t, theSave.Requests, "request1", `{"value":1}`) - assertHasValue(t, theSave.Requests, "request2", `{"value":2}`) - - assertLen(t, theSave.Imps, 1) - assertHasValue(t, theSave.Imps, "imp1", `{"value":1}`) -} - -func TestUpdates(t *testing.T) { - handler := &mockResponseHandler{ - statusCode: httpCore.StatusOK, - response: `{"requests":{"request1":{"value":1}, "request2":{"value":2}},"imps":{"imp1":{"value":3},"imp2":{"value":4}}}`, - } - server := httptest.NewServer(handler) - defer server.Close() - - ev := NewHTTPEvents(server.Client(), server.URL, nil, -1) - - handler.response = `{"requests":{"request1":{"value":5}, "request2":{"deleted":true}},"imps":{"imp1":{"deleted":true},"imp2":{"value":6}}}` - timeChan := make(chan time.Time, 1) - timeChan <- time.Now() - go ev.refresh(timeChan) - firstSave := <-ev.Saves() - secondSave := <-ev.Saves() - inv := <-ev.Invalidations() - - assertLen(t, firstSave.Requests, 2) - assertHasValue(t, firstSave.Requests, "request1", `{"value":1}`) - assertHasValue(t, firstSave.Requests, "request2", `{"value":2}`) - assertLen(t, firstSave.Imps, 2) - assertHasValue(t, firstSave.Imps, "imp1", `{"value":3}`) - assertHasValue(t, firstSave.Imps, "imp2", `{"value":4}`) - - assertLen(t, secondSave.Requests, 1) - assertHasValue(t, secondSave.Requests, "request1", `{"value":5}`) - assertLen(t, secondSave.Imps, 1) - assertHasValue(t, secondSave.Imps, "imp2", `{"value":6}`) - - assertArrLen(t, inv.Requests, 1) - assertArrContains(t, inv.Requests, "request2") - assertArrLen(t, inv.Imps, 1) - assertArrContains(t, inv.Imps, "imp1") -} -func TestErrorResponse(t *testing.T) { - handler := &mockResponseHandler{ - statusCode: httpCore.StatusInternalServerError, - response: "Something horrible happened.", - } - server := httptest.NewServer(handler) - defer server.Close() + "github.com/stretchr/testify/assert" +) - ev := NewHTTPEvents(server.Client(), server.URL, nil, -1) - if len(ev.Saves()) != 0 { - t.Errorf("No saves should be emitted if the HTTP call fails. Got %d", len(ev.Saves())) - } +func ctxProducer() (context.Context, func()) { + return context.WithTimeout(context.Background(), -1) } -func TestExpiredContext(t *testing.T) { - handler := &mockResponseHandler{ - statusCode: httpCore.StatusInternalServerError, - response: "Something horrible happened.", - } - server := httptest.NewServer(handler) - defer server.Close() - - ctxProducer := func() (context.Context, func()) { - return context.WithTimeout(context.Background(), -1) +func TestStartup(t *testing.T) { + type testStep struct { + statusCode int + response string + timeout bool + saves string + invalidations string } - - ev := NewHTTPEvents(server.Client(), server.URL, ctxProducer, -1) - if len(ev.Saves()) != 0 { - t.Errorf("No saves should be emitted if the HTTP call is cancelled. Got %d", len(ev.Saves())) - } -} - -func TestMalformedResponse(t *testing.T) { - handler := &mockResponseHandler{ - statusCode: httpCore.StatusOK, - response: "This isn't JSON.", + testCases := []struct { + description string + tests []testStep + }{ + { + description: "Load requests at startup", + tests: []testStep{ + { + statusCode: httpCore.StatusOK, + response: `{"requests": {"request1": {"value":1}, "request2": {"value":2}}}`, + saves: `{"requests": {"request1": {"value":1}, "request2": {"value":2}}, "imps": null, "accounts": null}`, + }, + }, + }, + { + description: "Load imps at startup", + tests: []testStep{ + { + statusCode: httpCore.StatusOK, + response: `{"imps": {"imp1": {"value":1}}}`, + saves: `{"imps": {"imp1": {"value":1}}, "requests": null, "accounts": null}`, + }, + }, + }, + { + description: "Load requests and imps then update", + tests: []testStep{ + { + statusCode: httpCore.StatusOK, + response: `{"requests": {"request1": {"value":1}, "request2": {"value":2}}, "imps": {"imp1": {"value":3}, "imp2": {"value":4}}}`, + saves: `{"requests": {"request1": {"value":1}, "request2": {"value":2}}, "imps": {"imp1": {"value":3}, "imp2": {"value":4}}, "accounts":null}`, + }, + { + statusCode: httpCore.StatusOK, + response: `{"requests": {"request1": {"value":5}, "request2": {"deleted":true}}, "imps": {"imp1": {"deleted":true}, "imp2": {"value":6}}}`, + saves: `{"requests": {"request1": {"value":5}}, "imps": {"imp2": {"value":6}}, "accounts":null}`, + invalidations: `{"requests": ["request2"], "imps": ["imp1"], "accounts": []}`, + }, + }, + }, + { + description: "Load accounts then update", + tests: []testStep{ + { + statusCode: httpCore.StatusOK, + response: `{"accounts":{"account1":{"value":1}, "account2":{"value":2}}}`, + saves: `{"accounts":{"account1":{"value":1}, "account2":{"value":2}}, "imps": null, "requests": null}`, + }, + { + statusCode: httpCore.StatusOK, + response: `{"accounts":{"account1":{"value":5}, "account2":{"deleted": true}}}`, + saves: `{"accounts":{"account1":{"value":5}}, "imps": null, "requests": null}`, + invalidations: `{"accounts":["account2"], "requests": [], "imps": []}`, + }, + }, + }, + { + description: "Load nothing at startup", + tests: []testStep{ + { + statusCode: httpCore.StatusOK, + response: `{}`, + }, + }, + }, + { + description: "Malformed response at startup", + tests: []testStep{ + { + statusCode: httpCore.StatusOK, + response: `{some bad json`, + }, + }, + }, + { + description: "Server error at startup", + tests: []testStep{ + { + statusCode: httpCore.StatusInternalServerError, + response: ``, + }, + }, + }, + { + description: "HTTP timeout error at startup", + tests: []testStep{ + { + timeout: true, + }, + }, + }, } - server := httptest.NewServer(handler) - defer server.Close() - - ev := NewHTTPEvents(server.Client(), server.URL, nil, -1) - if len(ev.Saves()) != 0 { - t.Errorf("No updates should be emitted if the HTTP call fails. Got %d", len(ev.Saves())) + for _, tests := range testCases { + t.Run(tests.description, func(t *testing.T) { + handler := &mockResponseHandler{} + server := httptest.NewServer(handler) + defer server.Close() + + var ev *HTTPEvents + + for i, test := range tests.tests { + handler.statusCode = test.statusCode + handler.response = test.response + if i == 0 { // NewHTTPEvents() calls the API immediately + if test.timeout { + ev = NewHTTPEvents(server.Client(), server.URL, ctxProducer, -1) // force timeout + } else { + ev = NewHTTPEvents(server.Client(), server.URL, nil, -1) + } + } else { // Second test triggers API call by initiating a 1s refresh loop + timeChan := make(chan time.Time, 1) + timeChan <- time.Now() + go ev.refresh(timeChan) + } + t.Run(fmt.Sprintf("Step %d", i+1), func(t *testing.T) { + // Check expected Saves + if len(test.saves) > 0 { + saves, err := json.Marshal(<-ev.Saves()) + assert.NoError(t, err, `Failed to marshal event.Save object: %v`, err) + assert.JSONEq(t, test.saves, string(saves)) + } + assert.Empty(t, ev.Saves(), "Unexpected additional messages in save channel") + // Check expected Invalidations + if len(test.invalidations) > 0 { + invalidations, err := json.Marshal(<-ev.Invalidations()) + assert.NoError(t, err, `Failed to marshal event.Invalidation object: %v`, err) + assert.JSONEq(t, test.invalidations, string(invalidations)) + } + assert.Empty(t, ev.Invalidations(), "Unexpected additional messages in invalidations channel") + }) + } + }) } } @@ -152,38 +168,3 @@ func (m *mockResponseHandler) ServeHTTP(rw httpCore.ResponseWriter, r *httpCore. rw.WriteHeader(m.statusCode) rw.Write([]byte(m.response)) } - -func assertLen(t *testing.T, m map[string]json.RawMessage, length int) { - t.Helper() - if len(m) != length { - t.Errorf("Expected map with %d elements, but got %v", length, m) - } -} - -func assertArrLen(t *testing.T, list []string, length int) { - t.Helper() - if len(list) != length { - t.Errorf("Expected list with %d elements, but got %v", length, list) - } -} - -func assertArrContains(t *testing.T, haystack []string, needle string) { - t.Helper() - for _, elm := range haystack { - if elm == needle { - return - } - } - t.Errorf("expected element %s to be in list %v", needle, haystack) -} - -func assertHasValue(t *testing.T, m map[string]json.RawMessage, key string, val string) { - t.Helper() - if mapVal, ok := m[key]; ok { - if !bytes.Equal(mapVal, []byte(val)) { - t.Errorf("expected map[%s] to be %s, but got %s", key, val, string(mapVal)) - } - } else { - t.Errorf("map missing expected key: %s", key) - } -} diff --git a/stored_requests/events/postgres/database.go b/stored_requests/events/postgres/database.go new file mode 100644 index 00000000000..54495ce42b0 --- /dev/null +++ b/stored_requests/events/postgres/database.go @@ -0,0 +1,225 @@ +package postgres + +import ( + "bytes" + "context" + "database/sql" + "encoding/json" + "net" + "time" + + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events" + "github.com/PubMatic-OpenWrap/prebid-server/util/timeutil" + "github.com/golang/glog" +) + +func bytesNull() []byte { + return []byte{'n', 'u', 'l', 'l'} +} + +var storedDataTypeMetricMap = map[config.DataType]pbsmetrics.StoredDataType{ + config.RequestDataType: pbsmetrics.RequestDataType, + config.CategoryDataType: pbsmetrics.CategoryDataType, + config.VideoDataType: pbsmetrics.VideoDataType, + config.AMPRequestDataType: pbsmetrics.AMPDataType, + config.AccountDataType: pbsmetrics.AccountDataType, +} + +type PostgresEventProducerConfig struct { + DB *sql.DB + RequestType config.DataType + CacheInitQuery string + CacheInitTimeout time.Duration + CacheUpdateQuery string + CacheUpdateTimeout time.Duration + MetricsEngine pbsmetrics.MetricsEngine +} + +type PostgresEventProducer struct { + cfg PostgresEventProducerConfig + lastUpdate time.Time + invalidations chan events.Invalidation + saves chan events.Save + time timeutil.Time +} + +func NewPostgresEventProducer(cfg PostgresEventProducerConfig) (eventProducer *PostgresEventProducer) { + if cfg.DB == nil { + glog.Fatalf("The Postgres Stored %s Loader needs a database connection to work.", cfg.RequestType) + } + + return &PostgresEventProducer{ + cfg: cfg, + lastUpdate: time.Time{}, + saves: make(chan events.Save, 1), + invalidations: make(chan events.Invalidation, 1), + time: &timeutil.RealTime{}, + } +} + +func (e *PostgresEventProducer) Run() error { + if e.lastUpdate.IsZero() { + return e.fetchAll() + } + + return e.fetchDelta() +} + +func (e *PostgresEventProducer) Saves() <-chan events.Save { + return e.saves +} + +func (e *PostgresEventProducer) Invalidations() <-chan events.Invalidation { + return e.invalidations +} + +func (e *PostgresEventProducer) fetchAll() (fetchErr error) { + timeout := e.cfg.CacheInitTimeout * time.Millisecond + ctx, cancel := context.WithTimeout(context.Background(), timeout) + defer cancel() + + startTime := e.time.Now().UTC() + rows, err := e.cfg.DB.QueryContext(ctx, e.cfg.CacheInitQuery) + elapsedTime := time.Since(startTime) + e.recordFetchTime(elapsedTime, pbsmetrics.FetchAll) + + if err != nil { + glog.Warningf("Failed to fetch all Stored %s data from the DB: %v", e.cfg.RequestType, err) + if _, ok := err.(net.Error); ok { + e.recordError(pbsmetrics.StoredDataErrorNetwork) + } else { + e.recordError(pbsmetrics.StoredDataErrorUndefined) + } + return err + } + + defer func() { + if err := rows.Close(); err != nil { + glog.Warningf("Failed to close the Stored %s DB connection: %v", e.cfg.RequestType, err) + e.recordError(pbsmetrics.StoredDataErrorUndefined) + fetchErr = err + } + }() + if err := e.sendEvents(rows); err != nil { + glog.Warningf("Failed to load all Stored %s data from the DB: %v", e.cfg.RequestType, err) + e.recordError(pbsmetrics.StoredDataErrorUndefined) + return err + } + + e.lastUpdate = startTime + return nil +} + +func (e *PostgresEventProducer) fetchDelta() (fetchErr error) { + timeout := e.cfg.CacheUpdateTimeout * time.Millisecond + ctx, cancel := context.WithTimeout(context.Background(), timeout) + defer cancel() + + startTime := e.time.Now().UTC() + rows, err := e.cfg.DB.QueryContext(ctx, e.cfg.CacheUpdateQuery, e.lastUpdate) + elapsedTime := time.Since(startTime) + e.recordFetchTime(elapsedTime, pbsmetrics.FetchDelta) + + if err != nil { + glog.Warningf("Failed to fetch updated Stored %s data from the DB: %v", e.cfg.RequestType, err) + if _, ok := err.(net.Error); ok { + e.recordError(pbsmetrics.StoredDataErrorNetwork) + } else { + e.recordError(pbsmetrics.StoredDataErrorUndefined) + } + return err + } + + defer func() { + if err := rows.Close(); err != nil { + glog.Warningf("Failed to close the Stored %s DB connection: %v", e.cfg.RequestType, err) + e.recordError(pbsmetrics.StoredDataErrorUndefined) + fetchErr = err + } + }() + if err := e.sendEvents(rows); err != nil { + glog.Warningf("Failed to load updated Stored %s data from the DB: %v", e.cfg.RequestType, err) + e.recordError(pbsmetrics.StoredDataErrorUndefined) + return err + } + + e.lastUpdate = startTime + return nil +} + +func (e *PostgresEventProducer) recordFetchTime(elapsedTime time.Duration, fetchType pbsmetrics.StoredDataFetchType) { + e.cfg.MetricsEngine.RecordStoredDataFetchTime( + pbsmetrics.StoredDataLabels{ + DataType: storedDataTypeMetricMap[e.cfg.RequestType], + DataFetchType: fetchType, + }, elapsedTime) +} + +func (e *PostgresEventProducer) recordError(errorType pbsmetrics.StoredDataError) { + e.cfg.MetricsEngine.RecordStoredDataError( + pbsmetrics.StoredDataLabels{ + DataType: storedDataTypeMetricMap[e.cfg.RequestType], + Error: errorType, + }) +} + +// sendEvents reads the rows and sends notifications into the channel for any updates. +// If it returns an error, then callers can be certain that no events were sent to the channels. +func (e *PostgresEventProducer) sendEvents(rows *sql.Rows) (err error) { + storedRequestData := make(map[string]json.RawMessage) + storedImpData := make(map[string]json.RawMessage) + + var requestInvalidations []string + var impInvalidations []string + + for rows.Next() { + var id string + var data []byte + var dataType string + + // discard corrupted data so it is not saved in the cache + if err := rows.Scan(&id, &data, &dataType); err != nil { + return err + } + + switch dataType { + case "request": + if len(data) == 0 || bytes.Equal(data, bytesNull()) { + requestInvalidations = append(requestInvalidations, id) + } else { + storedRequestData[id] = data + } + case "imp": + if len(data) == 0 || bytes.Equal(data, bytesNull()) { + impInvalidations = append(impInvalidations, id) + } else { + storedImpData[id] = data + } + default: + glog.Warningf("Stored Data with id=%s has invalid type: %s. This will be ignored.", id, dataType) + } + } + + // discard corrupted data so it is not saved in the cache + if rows.Err() != nil { + return rows.Err() + } + + if len(storedRequestData) > 0 || len(storedImpData) > 0 { + e.saves <- events.Save{ + Requests: storedRequestData, + Imps: storedImpData, + } + } + + if (len(requestInvalidations) > 0 || len(impInvalidations) > 0) && !e.lastUpdate.IsZero() { + e.invalidations <- events.Invalidation{ + Requests: requestInvalidations, + Imps: impInvalidations, + } + } + + return +} diff --git a/stored_requests/events/postgres/database_test.go b/stored_requests/events/postgres/database_test.go new file mode 100644 index 00000000000..c3a9b79c7b9 --- /dev/null +++ b/stored_requests/events/postgres/database_test.go @@ -0,0 +1,444 @@ +package postgres + +import ( + "encoding/json" + "errors" + "regexp" + "testing" + "time" + + "github.com/PubMatic-OpenWrap/prebid-server/config" + "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + + sqlmock "github.com/DATA-DOG/go-sqlmock" +) + +// FakeTime implements the Time interface +type FakeTime struct { + time time.Time +} + +func (mc *FakeTime) Now() time.Time { + return mc.time +} + +const fakeQuery = "SELECT id, requestData, type FROM stored_data" + +func fakeQueryRegex() string { + return "^" + regexp.QuoteMeta(fakeQuery) + "$" +} + +func TestFetchAllSuccess(t *testing.T) { + tests := []struct { + description string + giveFakeTime time.Time + giveMockRows *sqlmock.Rows + wantLastUpdate time.Time + wantSavedReqs map[string]json.RawMessage + wantSavedImps map[string]json.RawMessage + wantInvalidatedReqs []string + wantInvalidatedImps []string + }{ + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs = 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + }, + { + description: "saved reqs > 0, saved imps = 0, invalidated reqs = 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("req-1", "true", "request"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantSavedReqs: map[string]json.RawMessage{"req-1": json.RawMessage(`true`)}, + wantSavedImps: map[string]json.RawMessage{}, + }, + { + description: "saved reqs = 0, saved imps > 0, invalidated reqs = 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("imp-1", "true", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantSavedReqs: map[string]json.RawMessage{}, + wantSavedImps: map[string]json.RawMessage{"imp-1": json.RawMessage(`true`)}, + }, + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs > 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("req-1", "", "request"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + }, + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs = 0, invalidated imps > 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("imp-1", "", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + }, + { + description: "saved reqs > 0, saved imps > 0, invalidated reqs > 0, invalidated imps > 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}). + AddRow("req-1", "true", "request"). + AddRow("imp-1", "true", "imp"). + AddRow("req-2", "", "request"). + AddRow("imp-2", "", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantSavedReqs: map[string]json.RawMessage{"req-1": json.RawMessage(`true`)}, + wantSavedImps: map[string]json.RawMessage{"imp-1": json.RawMessage(`true`)}, + }, + } + + for _, tt := range tests { + db, dbMock, _ := sqlmock.New() + dbMock.ExpectQuery(fakeQueryRegex()).WillReturnRows(tt.giveMockRows) + + metricsMock := &pbsmetrics.MetricsEngineMock{} + metricsMock.Mock.On("RecordStoredDataFetchTime", pbsmetrics.StoredDataLabels{ + DataType: pbsmetrics.RequestDataType, + DataFetchType: pbsmetrics.FetchAll, + }, mock.Anything).Return() + + eventProducer := NewPostgresEventProducer(PostgresEventProducerConfig{ + DB: db, + RequestType: config.RequestDataType, + CacheInitTimeout: 100 * time.Millisecond, + CacheInitQuery: fakeQuery, + MetricsEngine: metricsMock, + }) + eventProducer.time = &FakeTime{time: tt.giveFakeTime} + err := eventProducer.Run() + + assert.Nil(t, err, tt.description) + assert.Equal(t, tt.wantLastUpdate, eventProducer.lastUpdate, tt.description) + + var saves events.Save + // Read data from saves channel with timeout to avoid test suite deadlock + select { + case saves = <-eventProducer.Saves(): + case <-time.After(20 * time.Millisecond): + } + var invalidations events.Invalidation + // Read data from invalidations channel with timeout to avoid test suite deadlock + select { + case invalidations = <-eventProducer.Invalidations(): + case <-time.After(20 * time.Millisecond): + } + + assert.Equal(t, tt.wantSavedReqs, saves.Requests, tt.description) + assert.Equal(t, tt.wantSavedImps, saves.Imps, tt.description) + assert.Equal(t, tt.wantInvalidatedReqs, invalidations.Requests, tt.description) + assert.Equal(t, tt.wantInvalidatedImps, invalidations.Imps, tt.description) + + metricsMock.AssertExpectations(t) + } +} + +func TestFetchAllErrors(t *testing.T) { + tests := []struct { + description string + giveFakeTime time.Time + giveTimeoutMS int + giveMockRows *sqlmock.Rows + wantRecordedError pbsmetrics.StoredDataError + wantLastUpdate time.Time + }{ + { + description: "fetch all timeout", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: nil, + wantRecordedError: pbsmetrics.StoredDataErrorNetwork, + wantLastUpdate: time.Time{}, + }, + { + description: "fetch all query error", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveTimeoutMS: 100, + giveMockRows: nil, + wantRecordedError: pbsmetrics.StoredDataErrorUndefined, + wantLastUpdate: time.Time{}, + }, + { + description: "fetch all row error", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveTimeoutMS: 100, + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}). + AddRow("stored-req-id", "true", "request"). + RowError(0, errors.New("Some row error.")), + wantRecordedError: pbsmetrics.StoredDataErrorUndefined, + wantLastUpdate: time.Time{}, + }, + } + + for _, tt := range tests { + db, dbMock, _ := sqlmock.New() + if tt.giveMockRows == nil { + dbMock.ExpectQuery(fakeQueryRegex()).WillReturnError(errors.New("Query failed.")) + } else { + dbMock.ExpectQuery(fakeQueryRegex()).WillReturnRows(tt.giveMockRows) + } + + metricsMock := &pbsmetrics.MetricsEngineMock{} + metricsMock.Mock.On("RecordStoredDataFetchTime", pbsmetrics.StoredDataLabels{ + DataType: pbsmetrics.RequestDataType, + DataFetchType: pbsmetrics.FetchAll, + }, mock.Anything).Return() + metricsMock.Mock.On("RecordStoredDataError", pbsmetrics.StoredDataLabels{ + DataType: pbsmetrics.RequestDataType, + Error: tt.wantRecordedError, + }).Return() + + eventProducer := NewPostgresEventProducer(PostgresEventProducerConfig{ + DB: db, + RequestType: config.RequestDataType, + CacheInitTimeout: time.Duration(tt.giveTimeoutMS) * time.Millisecond, + CacheInitQuery: fakeQuery, + MetricsEngine: metricsMock, + }) + eventProducer.time = &FakeTime{time: tt.giveFakeTime} + err := eventProducer.Run() + + assert.NotNil(t, err, tt.description) + assert.Equal(t, tt.wantLastUpdate, eventProducer.lastUpdate, tt.description) + + var saves events.Save + // Read data from saves channel with timeout to avoid test suite deadlock + select { + case saves = <-eventProducer.Saves(): + case <-time.After(10 * time.Millisecond): + } + var invalidations events.Invalidation + // Read data from invalidations channel with timeout to avoid test suite deadlock + select { + case invalidations = <-eventProducer.Invalidations(): + case <-time.After(10 * time.Millisecond): + } + + assert.Nil(t, saves.Requests, tt.description) + assert.Nil(t, saves.Imps, tt.description) + assert.Nil(t, invalidations.Requests, tt.description) + assert.Nil(t, invalidations.Requests, tt.description) + + metricsMock.AssertExpectations(t) + } +} + +func TestFetchDeltaSuccess(t *testing.T) { + tests := []struct { + description string + giveFakeTime time.Time + giveMockRows *sqlmock.Rows + wantLastUpdate time.Time + wantSavedReqs map[string]json.RawMessage + wantSavedImps map[string]json.RawMessage + wantInvalidatedReqs []string + wantInvalidatedImps []string + }{ + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs = 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + }, + { + description: "saved reqs > 0, saved imps = 0, invalidated reqs = 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("req-1", "true", "request"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantSavedReqs: map[string]json.RawMessage{"req-1": json.RawMessage(`true`)}, + wantSavedImps: map[string]json.RawMessage{}, + }, + { + description: "saved reqs = 0, saved imps > 0, invalidated reqs = 0, invalidated imps = 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("imp-1", "true", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantSavedReqs: map[string]json.RawMessage{}, + wantSavedImps: map[string]json.RawMessage{"imp-1": json.RawMessage(`true`)}, + }, + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs > 0, invalidated imps = 0, empty data", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("req-1", "", "request"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantInvalidatedReqs: []string{"req-1"}, + wantInvalidatedImps: nil, + }, + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs > 0, invalidated imps = 0, null data", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("req-1", "null", "request"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantInvalidatedReqs: []string{"req-1"}, + wantInvalidatedImps: nil, + }, + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs = 0, invalidated imps > 0, empty data", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("imp-1", "", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantInvalidatedImps: []string{"imp-1"}, + }, + { + description: "saved reqs = 0, saved imps = 0, invalidated reqs = 0, invalidated imps > 0, null data", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}).AddRow("imp-1", "null", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantInvalidatedImps: []string{"imp-1"}, + }, + { + description: "saved reqs > 0, saved imps > 0, invalidated reqs > 0, invalidated imps > 0", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}). + AddRow("req-1", "true", "request"). + AddRow("imp-1", "true", "imp"). + AddRow("req-2", "", "request"). + AddRow("imp-2", "", "imp"), + wantLastUpdate: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + wantSavedReqs: map[string]json.RawMessage{"req-1": json.RawMessage(`true`)}, + wantSavedImps: map[string]json.RawMessage{"imp-1": json.RawMessage(`true`)}, + wantInvalidatedReqs: []string{"req-2"}, + wantInvalidatedImps: []string{"imp-2"}, + }, + } + + for _, tt := range tests { + db, dbMock, _ := sqlmock.New() + dbMock.ExpectQuery(fakeQueryRegex()).WillReturnRows(tt.giveMockRows) + + metricsMock := &pbsmetrics.MetricsEngineMock{} + metricsMock.Mock.On("RecordStoredDataFetchTime", pbsmetrics.StoredDataLabels{ + DataType: pbsmetrics.RequestDataType, + DataFetchType: pbsmetrics.FetchDelta, + }, mock.Anything).Return() + + eventProducer := NewPostgresEventProducer(PostgresEventProducerConfig{ + DB: db, + RequestType: config.RequestDataType, + CacheUpdateTimeout: 100 * time.Millisecond, + CacheUpdateQuery: fakeQuery, + MetricsEngine: metricsMock, + }) + eventProducer.lastUpdate = time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC) + eventProducer.time = &FakeTime{time: tt.giveFakeTime} + err := eventProducer.Run() + + assert.Nil(t, err, tt.description) + assert.Equal(t, tt.wantLastUpdate, eventProducer.lastUpdate, tt.description) + + var saves events.Save + // Read data from saves channel with timeout to avoid test suite deadlock + select { + case saves = <-eventProducer.Saves(): + case <-time.After(20 * time.Millisecond): + } + var invalidations events.Invalidation + // Read data from invalidations channel with timeout to avoid test suite deadlock + select { + case invalidations = <-eventProducer.Invalidations(): + case <-time.After(20 * time.Millisecond): + } + + assert.Equal(t, tt.wantSavedReqs, saves.Requests, tt.description) + assert.Equal(t, tt.wantSavedImps, saves.Imps, tt.description) + assert.Equal(t, tt.wantInvalidatedReqs, invalidations.Requests, tt.description) + assert.Equal(t, tt.wantInvalidatedImps, invalidations.Imps, tt.description) + + metricsMock.AssertExpectations(t) + } +} + +func TestFetchDeltaErrors(t *testing.T) { + tests := []struct { + description string + giveFakeTime time.Time + giveTimeoutMS int + giveLastUpdate time.Time + giveMockRows *sqlmock.Rows + wantRecordedError pbsmetrics.StoredDataError + wantLastUpdate time.Time + }{ + { + description: "fetch delta timeout", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveLastUpdate: time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC), + giveMockRows: nil, + wantRecordedError: pbsmetrics.StoredDataErrorNetwork, + wantLastUpdate: time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC), + }, + { + description: "fetch delta query error", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveTimeoutMS: 100, + giveLastUpdate: time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC), + giveMockRows: nil, + wantRecordedError: pbsmetrics.StoredDataErrorUndefined, + wantLastUpdate: time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC), + }, + { + description: "fetch delta row error", + giveFakeTime: time.Date(2020, time.July, 1, 12, 30, 0, 0, time.UTC), + giveTimeoutMS: 100, + giveLastUpdate: time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC), + giveMockRows: sqlmock.NewRows([]string{"id", "data", "dataType"}). + AddRow("stored-req-id", "true", "request"). + RowError(0, errors.New("Some row error.")), + wantRecordedError: pbsmetrics.StoredDataErrorUndefined, + wantLastUpdate: time.Date(2020, time.June, 30, 6, 0, 0, 0, time.UTC), + }, + } + + for _, tt := range tests { + db, dbMock, _ := sqlmock.New() + if tt.giveMockRows == nil { + dbMock.ExpectQuery(fakeQueryRegex()).WillReturnError(errors.New("Query failed.")) + } else { + dbMock.ExpectQuery(fakeQueryRegex()).WillReturnRows(tt.giveMockRows) + } + + metricsMock := &pbsmetrics.MetricsEngineMock{} + metricsMock.Mock.On("RecordStoredDataFetchTime", pbsmetrics.StoredDataLabels{ + DataType: pbsmetrics.RequestDataType, + DataFetchType: pbsmetrics.FetchDelta, + }, mock.Anything).Return() + metricsMock.Mock.On("RecordStoredDataError", pbsmetrics.StoredDataLabels{ + DataType: pbsmetrics.RequestDataType, + Error: tt.wantRecordedError, + }).Return() + + eventProducer := NewPostgresEventProducer(PostgresEventProducerConfig{ + DB: db, + RequestType: config.RequestDataType, + CacheUpdateTimeout: time.Duration(tt.giveTimeoutMS) * time.Millisecond, + CacheUpdateQuery: fakeQuery, + MetricsEngine: metricsMock, + }) + eventProducer.lastUpdate = tt.giveLastUpdate + eventProducer.time = &FakeTime{time: tt.giveFakeTime} + err := eventProducer.Run() + + assert.NotNil(t, err, tt.description) + assert.Equal(t, tt.wantLastUpdate, eventProducer.lastUpdate, tt.description) + + var saves events.Save + // Read data from saves channel with timeout to avoid test suite deadlock + select { + case saves = <-eventProducer.Saves(): + case <-time.After(10 * time.Millisecond): + } + var invalidations events.Invalidation + // Read data from invalidations channel with timeout to avoid test suite deadlock + select { + case invalidations = <-eventProducer.Invalidations(): + case <-time.After(10 * time.Millisecond): + } + + assert.Nil(t, saves.Requests, tt.description) + assert.Nil(t, saves.Imps, tt.description) + assert.Nil(t, invalidations.Requests, tt.description) + assert.Nil(t, invalidations.Requests, tt.description) + + metricsMock.AssertExpectations(t) + } +} diff --git a/stored_requests/events/postgres/polling.go b/stored_requests/events/postgres/polling.go deleted file mode 100644 index f6d388ead70..00000000000 --- a/stored_requests/events/postgres/polling.go +++ /dev/null @@ -1,160 +0,0 @@ -package postgres - -import ( - "bytes" - "context" - "database/sql" - "encoding/json" - "time" - - "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events" - "github.com/golang/glog" -) - -// PollForUpdates returns an EventProducer which checks the database for updates every refreshRate. -// -// This object will prioritize thoroughness over efficiency. In rare cases it may produce two "update" events for -// the same DB save, but it should never "miss" a database update either. -// -// The Queries should return a ResultSet with the following columns and types: -// -// 1. id: string -// 2. data: JSON -// 3. type: string ("request" or "imp") -// -// If data is empty or the JSON "null", then the ID will be invalidated (e.g. a deletion). -// If data is not empty, it should be the Stored Request or Stored Imp data associated with the given ID. -func PollForUpdates(ctxProducer func() (ctx context.Context, canceller func()), db *sql.DB, query string, startUpdatesFrom time.Time, refreshRate time.Duration) (eventProducer *PostgresPoller) { - // If we're not given a function to produce Contexts, use the Background one. - if ctxProducer == nil { - ctxProducer = func() (ctx context.Context, canceller func()) { - return context.Background(), func() {} - } - } - if db == nil { - glog.Fatal("The Stored Request Postgres Poller needs a database connection to work.") - } - - e := &PostgresPoller{ - db: db, - ctxProducer: ctxProducer, - updateQuery: query, - lastUpdate: startUpdatesFrom, - invalidations: make(chan events.Invalidation, 1), - saves: make(chan events.Save, 1), - } - - glog.Infof("Stored Requests will be refreshed from Postgres every %f seconds with: %s", refreshRate.Seconds(), query) - - if refreshRate > 0 { - go e.refresh(time.Tick(refreshRate)) - } else { - glog.Warningf("Postgres Stored Event polling refreshRate was %d. This must be positive. No updates will occur.", refreshRate) - } - return e -} - -type PostgresPoller struct { - db *sql.DB - ctxProducer func() (ctx context.Context, canceller func()) - updateQuery string - lastUpdate time.Time - invalidations chan events.Invalidation - saves chan events.Save -} - -func (e *PostgresPoller) refresh(ticker <-chan time.Time) { - for { - select { - case thisTime := <-ticker: - // Make sure to log the time now, *before* running the query, - // so that next tick's query won't miss any new updates which were made at the same time. - // This may duplicate some updates, but safety > efficiency. - thisTimeInUTC := thisTime.UTC() - ctx, cancel := e.ctxProducer() - rows, err := e.db.QueryContext(ctx, e.updateQuery, e.lastUpdate) - if err != nil { - glog.Warningf("Failed to update Stored Request data: %v", err) - cancel() - continue - } - if err := sendEvents(rows, e.saves, e.invalidations); err != nil { - glog.Warningf("Failed to update Stored Request data: %v", err) - } else { - e.lastUpdate = thisTimeInUTC - } - if err := rows.Close(); err != nil { - glog.Warningf("Failed to close DB connection: %v", err) - } - cancel() - } - } -} - -// sendEvents reads the rows and sends notifications into the channel for any updates. -// If it returns an error, then callers can be certain that no events were sent to the channels. -func sendEvents(rows *sql.Rows, saves chan<- events.Save, invalidations chan<- events.Invalidation) (err error) { - storedRequestData := make(map[string]json.RawMessage) - storedImpData := make(map[string]json.RawMessage) - - var requestInvalidations []string - var impInvalidations []string - - for rows.Next() { - var id string - var data []byte - var dataType string - // Beware #338... we really don't want to save corrupt data - if err := rows.Scan(&id, &data, &dataType); err != nil { - return err - } - - switch dataType { - case "request": - if len(data) == 0 || bytes.Equal(data, []byte("null")) { - requestInvalidations = append(requestInvalidations, id) - } else { - storedRequestData[id] = data - } - case "imp": - if len(data) == 0 || bytes.Equal(data, []byte("null")) { - impInvalidations = append(impInvalidations, id) - } else { - storedImpData[id] = data - } - default: - glog.Warningf("Stored Data with id=%s has invalid type: %s. This will be ignored.", id, dataType) - } - } - - // Beware #338... we really don't want to save corrupt data - if rows.Err() != nil { - return rows.Err() - } - - if len(storedRequestData) > 0 || len(storedImpData) > 0 && saves != nil { - saves <- events.Save{ - Requests: storedRequestData, - Imps: storedImpData, - } - } - - // There shouldn't be any invalidations with a nil channel (a "startup" query), - // but... if there are, we certainly don't want to block forever. - if len(requestInvalidations) > 0 || len(impInvalidations) > 0 && invalidations != nil { - invalidations <- events.Invalidation{ - Requests: requestInvalidations, - Imps: impInvalidations, - } - } - - return nil -} - -func (e *PostgresPoller) Saves() <-chan events.Save { - return e.saves -} - -func (e *PostgresPoller) Invalidations() <-chan events.Invalidation { - return e.invalidations -} diff --git a/stored_requests/events/postgres/polling_test.go b/stored_requests/events/postgres/polling_test.go deleted file mode 100644 index 7dd7c325fe7..00000000000 --- a/stored_requests/events/postgres/polling_test.go +++ /dev/null @@ -1,65 +0,0 @@ -package postgres - -import ( - "regexp" - "testing" - "time" - - sqlmock "github.com/DATA-DOG/go-sqlmock" -) - -const updateQuery = "SELECT id, requestData, type FROM stored_data" - -func updateQueryRegex() string { - return "^" + regexp.QuoteMeta(updateQuery) + "$" -} - -func TestSuccessfulUpdates(t *testing.T) { - db, mock := newMock(t) - mockRows := sqlmock.NewRows([]string{"id", "data", "dataType"}). - AddRow("stored-req-1", "true", "request"). - AddRow("stored-req-2", "null", "request"). - AddRow("stored-imp-1", `{"id":1}`, "imp"). - AddRow("stored-imp-2", `{"id":2}`, "imp"). - AddRow("stored-imp-3", "", "imp") - - updateStart := time.Now() - - mock.ExpectQuery(initialQueryRegex()).WillReturnRows(mockRows) - - evs := PollForUpdates(nil, db, updateQuery, updateStart, time.Duration(-1)) - timeChan := make(chan time.Time) - go evs.refresh(timeChan) - timeChan <- time.Now() - - save := <-evs.Saves() - assertMapLength(t, 1, save.Requests) - assertMapValue(t, save.Requests, "stored-req-1", "true") - assertMapLength(t, 2, save.Imps) - assertMapValue(t, save.Imps, "stored-imp-1", `{"id":1}`) - assertMapValue(t, save.Imps, "stored-imp-2", `{"id":2}`) - - invalidate := <-evs.Invalidations() - assertNumInvalidations(t, 1, invalidate.Requests) - assertSliceContains(t, invalidate.Requests, "stored-req-2") - assertNumInvalidations(t, 1, invalidate.Imps) - assertSliceContains(t, invalidate.Imps, "stored-imp-3") -} - -func assertNumInvalidations(t *testing.T, expected int, vals []string) { - t.Helper() - - if len(vals) != expected { - t.Errorf("Expected %d invalidations. Got: %v", expected, vals) - } -} - -func assertSliceContains(t *testing.T, haystack []string, needle string) { - t.Helper() - for _, elm := range haystack { - if elm == needle { - return - } - } - t.Errorf("expected element %s to be in list %v", needle, haystack) -} diff --git a/stored_requests/events/postgres/startup.go b/stored_requests/events/postgres/startup.go deleted file mode 100644 index c65d117e78b..00000000000 --- a/stored_requests/events/postgres/startup.go +++ /dev/null @@ -1,61 +0,0 @@ -package postgres - -import ( - "context" - "database/sql" - - "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/events" - "github.com/golang/glog" -) - -// This function queries the database to get all the data, and is guaranteed to return -// an EventProducer with a single "events.Save" object already in the channel before returning. -// -// The string query should return Rows with the following columns and types: -// -// 1. id: string -// 2. data: JSON -// 3. type: string ("request" or "imp") -// -func LoadAll(ctx context.Context, db *sql.DB, query string) (eventProducer *PostgresLoader) { - if db == nil { - glog.Fatal("The Stored Request Postgres Startup needs a database connection to work.") - } - eventProducer = &PostgresLoader{ - saves: make(chan events.Save, 1), - } - eventProducer.doFetch(ctx, db, query) - return -} - -type PostgresLoader struct { - saves chan events.Save -} - -func (loader *PostgresLoader) doFetch(ctx context.Context, db *sql.DB, query string) { - glog.Infof("Loading all Stored Requests from Postgres with: %s", query) - rows, err := db.QueryContext(ctx, query) - if err != nil { - glog.Warningf("Failed to fetch Stored Requests from Postgres on startup. The app might be a bit slow to start. Error was: %v", err) - loader.saves <- events.Save{} - return - } - defer func() { - if err := rows.Close(); err != nil { - glog.Warningf("Failed to close DB connection: %v", err) - } - }() - - if err := sendEvents(rows, loader.saves, nil); err != nil { - glog.Warningf("Failed to fetch Stored Requests from Postgres on startup. Things might be a bit slow to start: %v", err) - loader.saves <- events.Save{} - } -} - -func (e *PostgresLoader) Saves() <-chan events.Save { - return e.saves -} - -func (e *PostgresLoader) Invalidations() <-chan events.Invalidation { - return nil -} diff --git a/stored_requests/events/postgres/startup_test.go b/stored_requests/events/postgres/startup_test.go deleted file mode 100644 index d0b99412b23..00000000000 --- a/stored_requests/events/postgres/startup_test.go +++ /dev/null @@ -1,119 +0,0 @@ -package postgres - -import ( - "bytes" - "context" - "database/sql" - "encoding/json" - "errors" - "regexp" - "testing" - - sqlmock "github.com/DATA-DOG/go-sqlmock" -) - -func TestSuccessfulFetch(t *testing.T) { - db, mock := newMock(t) - mockRows := sqlmock.NewRows([]string{"id", "data", "dataType"}). - AddRow("stored-req-id", "true", "request"). - AddRow("stored-imp-1", `{"id":1}`, "imp"). - AddRow("stored-imp-2", `{"id":2}`, "imp") - - mock.ExpectQuery(initialQueryRegex()).WillReturnRows(mockRows) - - evs := LoadAll(context.Background(), db, initialQuery) - save := <-evs.Saves() - assertMapLength(t, 1, save.Requests) - assertMapValue(t, save.Requests, "stored-req-id", "true") - - assertMapLength(t, 2, save.Imps) - assertMapValue(t, save.Imps, "stored-imp-1", `{"id":1}`) - assertMapValue(t, save.Imps, "stored-imp-2", `{"id":2}`) - assertExpectationsMet(t, mock) -} - -// Make sure that an empty save still gets sent on the channel if the SQL query fails. -func TestQueryError(t *testing.T) { - db, mock := newMock(t) - mock.ExpectQuery(initialQueryRegex()).WillReturnError(errors.New("Query failed.")) - - evs := LoadAll(context.Background(), db, initialQuery) - save := <-evs.Saves() - assertMapLength(t, 0, save.Requests) - assertMapLength(t, 0, save.Imps) - assertExpectationsMet(t, mock) -} - -func TestRowError(t *testing.T) { - db, mock := newMock(t) - mockRows := sqlmock.NewRows([]string{"id", "data", "dataType"}). - AddRow("stored-req-id", "true", "request"). - AddRow("stored-imp-1", `{"id":1}`, "imp"). - RowError(1, errors.New("Some row error.")) - mock.ExpectQuery(initialQueryRegex()).WillReturnRows(mockRows) - - evs := LoadAll(context.Background(), db, initialQuery) - save := <-evs.Saves() - assertMapLength(t, 0, save.Requests) - assertMapLength(t, 0, save.Imps) - assertExpectationsMet(t, mock) -} - -func TestRowCloseError(t *testing.T) { - db, mock := newMock(t) - mockRows := sqlmock.NewRows([]string{"id", "data", "dataType"}). - AddRow("stored-req-id", "true", "request"). - AddRow("stored-imp-id", `{"id":1}`, "imp"). - CloseError(errors.New("Failed to close rows.")) - mock.ExpectQuery(initialQueryRegex()).WillReturnRows(mockRows) - - evs := LoadAll(context.Background(), db, initialQuery) - save := <-evs.Saves() - assertMapLength(t, 1, save.Requests) - assertMapLength(t, 1, save.Imps) - assertExpectationsMet(t, mock) -} - -func newMock(t *testing.T) (db *sql.DB, mock sqlmock.Sqlmock) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("Failed to create mock: %v", err) - } - return -} - -const initialQuery = "SELECT id, requestData, type FROM stored_data" - -func initialQueryRegex() string { - return "^" + regexp.QuoteMeta(initialQuery) + "$" -} - -type result struct { - id string - data json.RawMessage - dataType string -} - -func assertMapLength(t *testing.T, expectedLen int, theMap map[string]json.RawMessage) { - t.Helper() - if len(theMap) != expectedLen { - t.Errorf("Wrong map length. Expected %d, Got %d.", expectedLen, len(theMap)) - } -} - -func assertMapValue(t *testing.T, m map[string]json.RawMessage, key string, val string) { - t.Helper() - if mapVal, ok := m[key]; ok { - if !bytes.Equal(mapVal, []byte(val)) { - t.Errorf("expected map[%s] to be %s, but got %s", key, val, string(mapVal)) - } - } else { - t.Errorf("map missing expected key: %s", key) - } -} - -func assertExpectationsMet(t *testing.T, mock sqlmock.Sqlmock) { - if err := mock.ExpectationsWereMet(); err != nil { - t.Errorf("sqlmock expectations were not met: %v", err) - } -} diff --git a/stored_requests/fetcher.go b/stored_requests/fetcher.go index d3dc44bb65b..b37de04a2ab 100644 --- a/stored_requests/fetcher.go +++ b/stored_requests/fetcher.go @@ -25,6 +25,11 @@ type Fetcher interface { FetchRequests(ctx context.Context, requestIDs []string, impIDs []string) (requestData map[string]json.RawMessage, impData map[string]json.RawMessage, errs []error) } +type AccountFetcher interface { + // FetchAccount fetches the host account configuration for a publisher + FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) +} + type CategoryFetcher interface { // FetchCategories fetches the ad-server/publisher specific category for the given IAB category FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) @@ -32,8 +37,9 @@ type CategoryFetcher interface { // AllFetcher is an interface that encapsulates both the original Fetcher and the CategoryFetcher type AllFetcher interface { - FetchRequests(ctx context.Context, requestIDs []string, impIDs []string) (requestData map[string]json.RawMessage, impData map[string]json.RawMessage, errs []error) - FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) + Fetcher + AccountFetcher + CategoryFetcher } // NotFoundError is an error type to flag that an ID was not found by the Fetcher. @@ -56,7 +62,12 @@ func (e NotFoundError) Error() string { // Cache is an intermediate layer which can be used to create more complex Fetchers by composition. // Implementations must be safe for concurrent access by multiple goroutines. // To add a Cache layer in front of a Fetcher, see WithCache() -type Cache interface { +type Cache struct { + Requests CacheJSON + Imps CacheJSON + Accounts CacheJSON +} +type CacheJSON interface { // Get works much like Fetcher.FetchRequests, with a few exceptions: // // 1. Any (actionable) errors should be logged by the implementation, rather than returned. @@ -67,37 +78,33 @@ type Cache interface { // // Nil slices and empty strings are treated as "no ops". That is, a nil requestID will always produce a nil // "stored request data" in the response. - Get(ctx context.Context, requestIDs []string, impIDs []string) (requestData map[string]json.RawMessage, impData map[string]json.RawMessage) + Get(ctx context.Context, ids []string) (data map[string]json.RawMessage) // Invalidate will ensure that all values associated with the given IDs // are no longer returned by the cache until new values are saved via Update - Invalidate(ctx context.Context, requestIDs []string, impIDs []string) + Invalidate(ctx context.Context, ids []string) // Save will add or overwrite the data in the cache at the given keys - Save(ctx context.Context, requestData map[string]json.RawMessage, impData map[string]json.RawMessage) + Save(ctx context.Context, data map[string]json.RawMessage) } // ComposedCache creates an interface to treat a slice of caches as a single cache -type ComposedCache []Cache +type ComposedCache []CacheJSON // Get will attempt to Get from the caches in the order in which they are in the slice, // stopping as soon as a value is found (or when all caches have been exhausted) -func (c ComposedCache) Get(ctx context.Context, requestIDs []string, impIDs []string) (requestData map[string]json.RawMessage, impData map[string]json.RawMessage) { - requestData = make(map[string]json.RawMessage, len(requestIDs)) - impData = make(map[string]json.RawMessage, len(impIDs)) +func (c ComposedCache) Get(ctx context.Context, ids []string) (data map[string]json.RawMessage) { + data = make(map[string]json.RawMessage, len(ids)) - remainingReqIDs := requestIDs - remainingImpIDs := impIDs + remainingIDs := ids for _, cache := range c { - cachedReqData, cachedImpData := cache.Get(ctx, remainingReqIDs, remainingImpIDs) - - requestData, remainingReqIDs = updateFromCache(requestData, remainingReqIDs, cachedReqData) - impData, remainingImpIDs = updateFromCache(impData, remainingImpIDs, cachedImpData) + cachedData := cache.Get(ctx, remainingIDs) + data, remainingIDs = updateFromCache(data, remainingIDs, cachedData) - // return if all ids filled - if len(remainingReqIDs) == 0 && len(remainingImpIDs) == 0 { - return + // finish early if all ids filled + if len(remainingIDs) == 0 { + break } } @@ -123,16 +130,16 @@ func updateFromCache(data map[string]json.RawMessage, ids []string, newData map[ } // Invalidate will propagate invalidations to all underlying caches -func (c ComposedCache) Invalidate(ctx context.Context, requestIDs []string, impIDs []string) { +func (c ComposedCache) Invalidate(ctx context.Context, ids []string) { for _, cache := range c { - cache.Invalidate(ctx, requestIDs, impIDs) + cache.Invalidate(ctx, ids) } } // Save will propagate saves to all underlying caches -func (c ComposedCache) Save(ctx context.Context, requestData map[string]json.RawMessage, impData map[string]json.RawMessage) { +func (c ComposedCache) Save(ctx context.Context, data map[string]json.RawMessage) { for _, cache := range c { - cache.Save(ctx, requestData, impData) + cache.Save(ctx, data) } } @@ -142,7 +149,7 @@ type fetcherWithCache struct { metricsEngine pbsmetrics.MetricsEngine } -// WithCache returns a Fetcher which uses the given Cache before delegating to the original. +// WithCache returns a Fetcher which uses the given Caches before delegating to the original. // This can be called multiple times to compose Cache layers onto the backing Fetcher, though // it is usually more desirable to first compose caches with Compose, ensuring propagation of updates // and invalidations through all cache layers. @@ -155,7 +162,9 @@ func WithCache(fetcher AllFetcher, cache Cache, metricsEngine pbsmetrics.Metrics } func (f *fetcherWithCache) FetchRequests(ctx context.Context, requestIDs []string, impIDs []string) (requestData map[string]json.RawMessage, impData map[string]json.RawMessage, errs []error) { - requestData, impData = f.cache.Get(ctx, requestIDs, impIDs) + + requestData = f.cache.Requests.Get(ctx, requestIDs) + impData = f.cache.Imps.Get(ctx, impIDs) // Fixes #311 leftoverImps := findLeftovers(impIDs, impData) @@ -172,7 +181,8 @@ func (f *fetcherWithCache) FetchRequests(ctx context.Context, requestIDs []strin fetcherReqData, fetcherImpData, fetcherErrs := f.fetcher.FetchRequests(ctx, leftoverReqs, leftoverImps) errs = fetcherErrs - f.cache.Save(ctx, fetcherReqData, fetcherImpData) + f.cache.Requests.Save(ctx, fetcherReqData) + f.cache.Imps.Save(ctx, fetcherImpData) requestData = mergeData(requestData, fetcherReqData) impData = mergeData(impData, fetcherImpData) @@ -181,6 +191,22 @@ func (f *fetcherWithCache) FetchRequests(ctx context.Context, requestIDs []strin return } +func (f *fetcherWithCache) FetchAccount(ctx context.Context, accountID string) (account json.RawMessage, errs []error) { + accountData := f.cache.Accounts.Get(ctx, []string{accountID}) + // TODO: add metrics + if account, ok := accountData[accountID]; ok { + f.metricsEngine.RecordAccountCacheResult(pbsmetrics.CacheHit, 1) + return account, errs + } else { + f.metricsEngine.RecordAccountCacheResult(pbsmetrics.CacheMiss, 1) + } + account, errs = f.fetcher.FetchAccount(ctx, accountID) + if len(errs) == 0 { + f.cache.Accounts.Save(ctx, map[string]json.RawMessage{accountID: account}) + } + return account, errs +} + func (f *fetcherWithCache) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { return "", nil } diff --git a/stored_requests/fetcher_test.go b/stored_requests/fetcher_test.go index 2e505d35a88..6285542fd85 100644 --- a/stored_requests/fetcher_test.go +++ b/stored_requests/fetcher_test.go @@ -7,30 +7,33 @@ import ( "testing" "github.com/PubMatic-OpenWrap/prebid-server/pbsmetrics" + "github.com/PubMatic-OpenWrap/prebid-server/stored_requests/caches/nil_cache" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" ) -func setupFetcherWithCacheDeps() (*mockCache, *mockFetcher, AllFetcher, *pbsmetrics.MetricsEngineMock) { - cache := &mockCache{} +func setupFetcherWithCacheDeps() (*mockCache, *mockCache, *mockFetcher, AllFetcher, *pbsmetrics.MetricsEngineMock) { + reqCache := &mockCache{} + impCache := &mockCache{} metricsEngine := &pbsmetrics.MetricsEngineMock{} fetcher := &mockFetcher{} - afetcherWithCache := WithCache(fetcher, cache, metricsEngine) + afetcherWithCache := WithCache(fetcher, Cache{reqCache, impCache, &nil_cache.NilCache{}}, metricsEngine) - return cache, fetcher, afetcherWithCache, metricsEngine + return reqCache, impCache, fetcher, afetcherWithCache, metricsEngine } func TestPerfectCache(t *testing.T) { - cache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() + reqCache, impCache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() impIDs := []string{"known"} reqIDs := []string{"req-id"} ctx := context.Background() - cache.On("Get", ctx, reqIDs, impIDs).Return( + reqCache.On("Get", ctx, reqIDs).Return( map[string]json.RawMessage{ "req-id": json.RawMessage(`{"req":true}`), - }, + }) + impCache.On("Get", ctx, impIDs).Return( map[string]json.RawMessage{ "known": json.RawMessage(`{}`), }) @@ -41,7 +44,8 @@ func TestPerfectCache(t *testing.T) { reqData, impData, errs := aFetcherWithCache.FetchRequests(ctx, reqIDs, impIDs) - cache.AssertExpectations(t) + reqCache.AssertExpectations(t) + impCache.AssertExpectations(t) fetcher.AssertExpectations(t) metricsEngine.AssertExpectations(t) assert.JSONEq(t, `{"req":true}`, string(reqData["req-id"]), "Fetch requests should fetch the right request data") @@ -50,15 +54,16 @@ func TestPerfectCache(t *testing.T) { } func TestImperfectCache(t *testing.T) { - cache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() + reqCache, impCache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() impIDs := []string{"cached", "uncached"} ctx := context.Background() - cache.On("Get", ctx, []string(nil), impIDs).Return( - map[string]json.RawMessage{}, + impCache.On("Get", ctx, impIDs).Return( map[string]json.RawMessage{ "cached": json.RawMessage(`true`), }) + reqCache.On("Get", ctx, []string(nil)).Return( + map[string]json.RawMessage{}) fetcher.On("FetchRequests", ctx, []string{}, []string{"uncached"}).Return( map[string]json.RawMessage{}, @@ -67,11 +72,11 @@ func TestImperfectCache(t *testing.T) { }, []error{}, ) - cache.On("Save", ctx, - map[string]json.RawMessage{}, + impCache.On("Save", ctx, map[string]json.RawMessage{ "uncached": json.RawMessage(`false`), }) + reqCache.On("Save", ctx, map[string]json.RawMessage{}) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheHit, 0) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheMiss, 0) metricsEngine.On("RecordStoredImpCacheResult", pbsmetrics.CacheHit, 1) @@ -79,7 +84,7 @@ func TestImperfectCache(t *testing.T) { reqData, impData, errs := aFetcherWithCache.FetchRequests(ctx, nil, impIDs) - cache.AssertExpectations(t) + impCache.AssertExpectations(t) fetcher.AssertExpectations(t) metricsEngine.AssertExpectations(t) assert.Len(t, reqData, 0, "Fetch requests should return nil if no request IDs were passed") @@ -89,14 +94,15 @@ func TestImperfectCache(t *testing.T) { } func TestMissingData(t *testing.T) { - cache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() + reqCache, impCache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() impIDs := []string{"unknown"} ctx := context.Background() - cache.On("Get", ctx, []string(nil), impIDs).Return( - map[string]json.RawMessage{}, + impCache.On("Get", ctx, impIDs).Return( map[string]json.RawMessage{}, ) + reqCache.On("Get", ctx, []string(nil)).Return( + map[string]json.RawMessage{}) fetcher.On("FetchRequests", ctx, []string{}, impIDs).Return( map[string]json.RawMessage{}, map[string]json.RawMessage{}, @@ -104,8 +110,10 @@ func TestMissingData(t *testing.T) { errors.New("Data not found"), }, ) - cache.On("Save", ctx, + impCache.On("Save", ctx, map[string]json.RawMessage{}, + ) + reqCache.On("Save", ctx, map[string]json.RawMessage{}, ) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheHit, 0) @@ -115,7 +123,8 @@ func TestMissingData(t *testing.T) { reqData, impData, errs := aFetcherWithCache.FetchRequests(ctx, nil, impIDs) - cache.AssertExpectations(t) + reqCache.AssertExpectations(t) + impCache.AssertExpectations(t) fetcher.AssertExpectations(t) metricsEngine.AssertExpectations(t) assert.Len(t, errs, 1, "FetchRequests for missing data should return an error") @@ -125,15 +134,16 @@ func TestMissingData(t *testing.T) { // Prevents #311 func TestCacheSaves(t *testing.T) { - cache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() + reqCache, impCache, fetcher, aFetcherWithCache, metricsEngine := setupFetcherWithCacheDeps() impIDs := []string{"abc", "abc"} ctx := context.Background() - cache.On("Get", ctx, []string(nil), impIDs).Return( - map[string]json.RawMessage{}, + impCache.On("Get", ctx, impIDs).Return( map[string]json.RawMessage{ "abc": json.RawMessage(`{}`), }) + reqCache.On("Get", ctx, []string(nil)).Return( + map[string]json.RawMessage{}) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheHit, 0) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheMiss, 0) metricsEngine.On("RecordStoredImpCacheResult", pbsmetrics.CacheHit, 2) @@ -141,7 +151,7 @@ func TestCacheSaves(t *testing.T) { _, impData, errs := aFetcherWithCache.FetchRequests(ctx, nil, []string{"abc", "abc"}) - cache.AssertExpectations(t) + impCache.AssertExpectations(t) fetcher.AssertExpectations(t) metricsEngine.AssertExpectations(t) assert.Len(t, impData, 1, "FetchRequests should return data only once for duplicate requests") @@ -149,43 +159,92 @@ func TestCacheSaves(t *testing.T) { assert.Len(t, errs, 0, "FetchRequests with duplicate IDs shouldn't return an error") } +func setupAccountFetcherWithCacheDeps() (*mockCache, *mockFetcher, AllFetcher, *pbsmetrics.MetricsEngineMock) { + accCache := &mockCache{} + metricsEngine := &pbsmetrics.MetricsEngineMock{} + fetcher := &mockFetcher{} + afetcherWithCache := WithCache(fetcher, Cache{&nil_cache.NilCache{}, &nil_cache.NilCache{}, accCache}, metricsEngine) + + return accCache, fetcher, afetcherWithCache, metricsEngine +} + +func TestAccountCacheHit(t *testing.T) { + accCache, fetcher, aFetcherWithCache, metricsEngine := setupAccountFetcherWithCacheDeps() + cachedAccounts := []string{"known"} + ctx := context.Background() + + // Test read from cache + accCache.On("Get", ctx, cachedAccounts).Return( + map[string]json.RawMessage{ + "known": json.RawMessage(`true`), + }) + + metricsEngine.On("RecordAccountCacheResult", pbsmetrics.CacheHit, 1) + account, errs := aFetcherWithCache.FetchAccount(ctx, "known") + + accCache.AssertExpectations(t) + fetcher.AssertExpectations(t) + metricsEngine.AssertExpectations(t) + assert.JSONEq(t, `true`, string(account), "FetchAccount should fetch the right account data") + assert.Len(t, errs, 0, "FetchAccount shouldn't return any errors") +} + +func TestAccountCacheMiss(t *testing.T) { + accCache, fetcher, aFetcherWithCache, metricsEngine := setupAccountFetcherWithCacheDeps() + uncachedAccounts := []string{"uncached"} + uncachedAccountsData := map[string]json.RawMessage{ + "uncached": json.RawMessage(`true`), + } + ctx := context.Background() + + // Test read from cache + accCache.On("Get", ctx, uncachedAccounts).Return(map[string]json.RawMessage{}) + accCache.On("Save", ctx, uncachedAccountsData) + fetcher.On("FetchAccount", ctx, "uncached").Return(uncachedAccountsData["uncached"], []error{}) + metricsEngine.On("RecordAccountCacheResult", pbsmetrics.CacheMiss, 1) + + account, errs := aFetcherWithCache.FetchAccount(ctx, "uncached") + + accCache.AssertExpectations(t) + fetcher.AssertExpectations(t) + metricsEngine.AssertExpectations(t) + assert.JSONEq(t, `true`, string(account), "FetchAccount should fetch the right account data") + assert.Len(t, errs, 0, "FetchAccount shouldn't return any errors") +} + func TestComposedCache(t *testing.T) { c1 := &mockCache{} c2 := &mockCache{} c3 := &mockCache{} c4 := &mockCache{} - cache := ComposedCache{c1, c2, c3, c4} + impCache := &mockCache{} + cache := Cache{ + Requests: ComposedCache{c1, c2, c3, c4}, + Imps: impCache, + } metricsEngine := &pbsmetrics.MetricsEngineMock{} fetcher := &mockFetcher{} aFetcherWithCache := WithCache(fetcher, cache, metricsEngine) - impIDs := []string{"1", "2", "3"} reqIDs := []string{"1", "2", "3"} + impIDs := []string{} ctx := context.Background() - c1.On("Get", ctx, reqIDs, impIDs).Return( - map[string]json.RawMessage{ - "1": json.RawMessage(`{"id": "1"}`), - }, + c1.On("Get", ctx, reqIDs).Return( map[string]json.RawMessage{ "1": json.RawMessage(`{"id": "1"}`), }) - c2.On("Get", ctx, []string{"2", "3"}, []string{"2", "3"}).Return( - map[string]json.RawMessage{ - "2": json.RawMessage(`{"id": "2"}`), - }, + c2.On("Get", ctx, []string{"2", "3"}).Return( map[string]json.RawMessage{ "2": json.RawMessage(`{"id": "2"}`), }) - c3.On("Get", ctx, []string{"3"}, []string{"3"}).Return( - map[string]json.RawMessage{ - "3": json.RawMessage(`{"id": "3"}`), - }, + c3.On("Get", ctx, []string{"3"}).Return( map[string]json.RawMessage{ "3": json.RawMessage(`{"id": "3"}`), }) + impCache.On("Get", ctx, []string{}).Return(map[string]json.RawMessage{}) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheHit, 3) metricsEngine.On("RecordStoredReqCacheResult", pbsmetrics.CacheMiss, 0) - metricsEngine.On("RecordStoredImpCacheResult", pbsmetrics.CacheHit, 3) + metricsEngine.On("RecordStoredImpCacheResult", pbsmetrics.CacheHit, 0) metricsEngine.On("RecordStoredImpCacheResult", pbsmetrics.CacheMiss, 0) reqData, impData, errs := aFetcherWithCache.FetchRequests(ctx, reqIDs, impIDs) @@ -193,14 +252,12 @@ func TestComposedCache(t *testing.T) { c1.AssertExpectations(t) c2.AssertExpectations(t) c3.AssertExpectations(t) + impCache.AssertExpectations(t) fetcher.AssertExpectations(t) metricsEngine.AssertExpectations(t) assert.Len(t, reqData, len(reqIDs), "FetchRequests should be able to return all request data from a composed cache") assert.Len(t, impData, len(impIDs), "FetchRequests should be able to return all imp data from a composed cache") assert.Len(t, errs, 0, "FetchRequests shouldn't return an error when trying to use a composed cache") - assert.JSONEq(t, `{"id": "1"}`, string(impData["1"]), "FetchRequests should fetch the right imp data") - assert.JSONEq(t, `{"id": "2"}`, string(impData["2"]), "FetchRequests should fetch the right imp data") - assert.JSONEq(t, `{"id": "3"}`, string(impData["3"]), "FetchRequests should fetch the right imp data") assert.JSONEq(t, `{"id": "1"}`, string(reqData["1"]), "FetchRequests should fetch the right req data") assert.JSONEq(t, `{"id": "2"}`, string(reqData["2"]), "FetchRequests should fetch the right req data") assert.JSONEq(t, `{"id": "3"}`, string(reqData["3"]), "FetchRequests should fetch the right req data") @@ -215,6 +272,11 @@ func (f *mockFetcher) FetchRequests(ctx context.Context, requestIDs []string, im return args.Get(0).(map[string]json.RawMessage), args.Get(1).(map[string]json.RawMessage), args.Get(2).([]error) } +func (a *mockFetcher) FetchAccount(ctx context.Context, accountID string) (json.RawMessage, []error) { + args := a.Called(ctx, accountID) + return args.Get(0).(json.RawMessage), args.Get(1).([]error) +} + func (f *mockFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { return "", nil } @@ -223,15 +285,15 @@ type mockCache struct { mock.Mock } -func (c *mockCache) Get(ctx context.Context, requestIDs []string, impIDs []string) (map[string]json.RawMessage, map[string]json.RawMessage) { - args := c.Called(ctx, requestIDs, impIDs) - return args.Get(0).(map[string]json.RawMessage), args.Get(1).(map[string]json.RawMessage) +func (c *mockCache) Get(ctx context.Context, ids []string) map[string]json.RawMessage { + args := c.Called(ctx, ids) + return args.Get(0).(map[string]json.RawMessage) } -func (c *mockCache) Save(ctx context.Context, storedRequests map[string]json.RawMessage, storedImps map[string]json.RawMessage) { - c.Called(ctx, storedRequests, storedImps) +func (c *mockCache) Save(ctx context.Context, data map[string]json.RawMessage) { + c.Called(ctx, data) } -func (c *mockCache) Invalidate(ctx context.Context, requestIDs []string, impIDs []string) { - c.Called(ctx, requestIDs, impIDs) +func (c *mockCache) Invalidate(ctx context.Context, ids []string) { + c.Called(ctx, ids) } diff --git a/stored_requests/multifetcher.go b/stored_requests/multifetcher.go index 24cf848448c..2d08fd45337 100644 --- a/stored_requests/multifetcher.go +++ b/stored_requests/multifetcher.go @@ -36,6 +36,21 @@ func (mf MultiFetcher) FetchRequests(ctx context.Context, requestIDs []string, i return } +func (mf MultiFetcher) FetchAccount(ctx context.Context, accountID string) (account json.RawMessage, errs []error) { + for _, f := range mf { + if af, ok := f.(AccountFetcher); ok { + if account, accErrs := af.FetchAccount(ctx, accountID); len(accErrs) == 0 { + return account, nil + } else { + accErrs = dropMissingIDs(accErrs) + errs = append(errs, accErrs...) + } + } + } + errs = append(errs, NotFoundError{accountID, "Account"}) + return nil, errs +} + func (mf MultiFetcher) FetchCategories(ctx context.Context, primaryAdServer, publisherId, iabCategory string) (string, error) { for _, f := range mf { if cf, ok := f.(CategoryFetcher); ok { diff --git a/stored_requests/multifetcher_test.go b/stored_requests/multifetcher_test.go index e703c2c9dcc..5035cfba82e 100644 --- a/stored_requests/multifetcher_test.go +++ b/stored_requests/multifetcher_test.go @@ -125,3 +125,54 @@ func TestOtherError(t *testing.T) { assert.JSONEq(t, `{"req_id": "def"}`, string(reqData["def"]), "MultiFetcher should return the right request data") assert.JSONEq(t, `{"imp_id": "imp-1"}`, string(impData["imp-1"]), "MultiFetcher should return the right imp data") } + +func TestMultiFetcherAccountFoundInFirstFetcher(t *testing.T) { + f1 := &mockFetcher{} + f2 := &mockFetcher{} + fetcher := &MultiFetcher{f1, f2} + ctx := context.Background() + + f1.On("FetchAccount", ctx, "ONE").Once().Return(json.RawMessage(`{"id": "ONE"}`), []error{}) + + account, errs := fetcher.FetchAccount(ctx, "ONE") + + f1.AssertExpectations(t) + f2.AssertNotCalled(t, "FetchAccount") + assert.Empty(t, errs) + assert.JSONEq(t, `{"id": "ONE"}`, string(account)) +} + +func TestMultiFetcherAccountFoundInSecondFetcher(t *testing.T) { + f1 := &mockFetcher{} + f2 := &mockFetcher{} + fetcher := &MultiFetcher{f1, f2} + ctx := context.Background() + + f1.On("FetchAccount", ctx, "TWO").Once().Return(json.RawMessage(``), []error{NotFoundError{"TWO", "Account"}}) + f2.On("FetchAccount", ctx, "TWO").Once().Return(json.RawMessage(`{"id": "TWO"}`), []error{}) + + account, errs := fetcher.FetchAccount(ctx, "TWO") + + f1.AssertExpectations(t) + f2.AssertExpectations(t) + assert.Empty(t, errs) + assert.JSONEq(t, `{"id": "TWO"}`, string(account)) +} + +func TestMultiFetcherAccountNotFound(t *testing.T) { + f1 := &mockFetcher{} + f2 := &mockFetcher{} + fetcher := &MultiFetcher{f1, f2} + ctx := context.Background() + + f1.On("FetchAccount", ctx, "MISSING").Once().Return(json.RawMessage(``), []error{NotFoundError{"TWO", "Account"}}) + f2.On("FetchAccount", ctx, "MISSING").Once().Return(json.RawMessage(``), []error{NotFoundError{"TWO", "Account"}}) + + account, errs := fetcher.FetchAccount(ctx, "MISSING") + + f1.AssertExpectations(t) + f2.AssertExpectations(t) + assert.Len(t, errs, 1) + assert.Nil(t, account) + assert.EqualError(t, errs[0], NotFoundError{"MISSING", "Account"}.Error()) +} diff --git a/usersync/usersyncers/syncer.go b/usersync/usersyncers/syncer.go index aaead65de33..482d7ba0286 100755 --- a/usersync/usersyncers/syncer.go +++ b/usersync/usersyncers/syncer.go @@ -5,9 +5,11 @@ import ( "text/template" ttx "github.com/PubMatic-OpenWrap/prebid-server/adapters/33across" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/acuityads" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adform" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adkernel" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adkernelAdn" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/adman" "github.com/PubMatic-OpenWrap/prebid-server/adapters/admixer" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adocean" "github.com/PubMatic-OpenWrap/prebid-server/adapters/adpone" @@ -15,12 +17,15 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/adapters/adtelligent" "github.com/PubMatic-OpenWrap/prebid-server/adapters/advangelists" "github.com/PubMatic-OpenWrap/prebid-server/adapters/aja" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/amx" "github.com/PubMatic-OpenWrap/prebid-server/adapters/appnexus" "github.com/PubMatic-OpenWrap/prebid-server/adapters/audienceNetwork" "github.com/PubMatic-OpenWrap/prebid-server/adapters/avocet" "github.com/PubMatic-OpenWrap/prebid-server/adapters/beachfront" "github.com/PubMatic-OpenWrap/prebid-server/adapters/beintoo" "github.com/PubMatic-OpenWrap/prebid-server/adapters/brightroll" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/colossus" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/connectad" "github.com/PubMatic-OpenWrap/prebid-server/adapters/consumable" "github.com/PubMatic-OpenWrap/prebid-server/adapters/conversant" "github.com/PubMatic-OpenWrap/prebid-server/adapters/cpmstar" @@ -34,14 +39,18 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/adapters/grid" "github.com/PubMatic-OpenWrap/prebid-server/adapters/gumgum" "github.com/PubMatic-OpenWrap/prebid-server/adapters/improvedigital" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/invibes" "github.com/PubMatic-OpenWrap/prebid-server/adapters/ix" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/krushmedia" "github.com/PubMatic-OpenWrap/prebid-server/adapters/lifestreet" "github.com/PubMatic-OpenWrap/prebid-server/adapters/lockerdome" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/logicad" "github.com/PubMatic-OpenWrap/prebid-server/adapters/lunamedia" "github.com/PubMatic-OpenWrap/prebid-server/adapters/marsmedia" "github.com/PubMatic-OpenWrap/prebid-server/adapters/mgid" "github.com/PubMatic-OpenWrap/prebid-server/adapters/nanointeractive" "github.com/PubMatic-OpenWrap/prebid-server/adapters/ninthdecimal" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/nobid" "github.com/PubMatic-OpenWrap/prebid-server/adapters/openx" "github.com/PubMatic-OpenWrap/prebid-server/adapters/pubmatic" "github.com/PubMatic-OpenWrap/prebid-server/adapters/pulsepoint" @@ -49,7 +58,9 @@ import ( "github.com/PubMatic-OpenWrap/prebid-server/adapters/rtbhouse" "github.com/PubMatic-OpenWrap/prebid-server/adapters/rubicon" "github.com/PubMatic-OpenWrap/prebid-server/adapters/sharethrough" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/smartadserver" "github.com/PubMatic-OpenWrap/prebid-server/adapters/smartrtb" + "github.com/PubMatic-OpenWrap/prebid-server/adapters/smartyads" "github.com/PubMatic-OpenWrap/prebid-server/adapters/somoaudience" "github.com/PubMatic-OpenWrap/prebid-server/adapters/sonobi" "github.com/PubMatic-OpenWrap/prebid-server/adapters/sovrn" @@ -80,9 +91,11 @@ func NewSyncerMap(cfg *config.Configuration) map[openrtb_ext.BidderName]usersync syncers := make(map[openrtb_ext.BidderName]usersync.Usersyncer, len(cfg.Adapters)) insertIntoMap(cfg, syncers, openrtb_ext.Bidder33Across, ttx.New33AcrossSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderAcuityAds, acuityads.NewAcuityAdsSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdform, adform.NewAdformSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdkernel, adkernel.NewAdkernelSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdkernelAdn, adkernelAdn.NewAdkernelAdnSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderAdman, adman.NewAdmanSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdmixer, admixer.NewAdmixerSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdOcean, adocean.NewAdOceanSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdpone, adpone.NewadponeSyncer) @@ -90,11 +103,14 @@ func NewSyncerMap(cfg *config.Configuration) map[openrtb_ext.BidderName]usersync insertIntoMap(cfg, syncers, openrtb_ext.BidderAdtelligent, adtelligent.NewAdtelligentSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAdvangelists, advangelists.NewAdvangelistsSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAJA, aja.NewAJASyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderAMX, amx.NewAMXSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAppnexus, appnexus.NewAppnexusSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderAvocet, avocet.NewAvocetSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderBeachfront, beachfront.NewBeachfrontSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderBeintoo, beintoo.NewBeintooSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderBrightroll, brightroll.NewBrightrollSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderColossus, colossus.NewColossusSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderConnectAd, connectad.NewConnectAdSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderConsumable, consumable.NewConsumableSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderConversant, conversant.NewConversantSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderCpmstar, cpmstar.NewCpmstarSyncer) @@ -109,14 +125,18 @@ func NewSyncerMap(cfg *config.Configuration) map[openrtb_ext.BidderName]usersync insertIntoMap(cfg, syncers, openrtb_ext.BidderGrid, grid.NewGridSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderGumGum, gumgum.NewGumGumSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderImprovedigital, improvedigital.NewImprovedigitalSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderInvibes, invibes.NewInvibesSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderIx, ix.NewIxSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderKrushmedia, krushmedia.NewKrushmediaSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderLifestreet, lifestreet.NewLifestreetSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderLockerDome, lockerdome.NewLockerDomeSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderLogicad, logicad.NewLogicadSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderLunaMedia, lunamedia.NewLunaMediaSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderMarsmedia, marsmedia.NewMarsmediaSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderMgid, mgid.NewMgidSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderNanoInteractive, nanointeractive.NewNanoInteractiveSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderNinthDecimal, ninthdecimal.NewNinthDecimalSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderNoBid, nobid.NewNoBidSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderOpenx, openx.NewOpenxSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderPubmatic, pubmatic.NewPubmaticSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderPulsepoint, pulsepoint.NewPulsepointSyncer) @@ -127,7 +147,9 @@ func NewSyncerMap(cfg *config.Configuration) map[openrtb_ext.BidderName]usersync insertIntoMap(cfg, syncers, openrtb_ext.BidderSomoaudience, somoaudience.NewSomoaudienceSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderSonobi, sonobi.NewSonobiSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderSovrn, sovrn.NewSovrnSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderSmartadserver, smartadserver.NewSmartadserverSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderSmartRTB, smartrtb.NewSmartRTBSyncer) + insertIntoMap(cfg, syncers, openrtb_ext.BidderSmartyAds, smartyads.NewSmartyAdsSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderSynacormedia, synacormedia.NewSynacorMediaSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderTelaria, telaria.NewTelariaSyncer) insertIntoMap(cfg, syncers, openrtb_ext.BidderTriplelift, triplelift.NewTripleliftSyncer) diff --git a/usersync/usersyncers/syncer_test.go b/usersync/usersyncers/syncer_test.go index 0bc2f6a458d..7833605ea76 100755 --- a/usersync/usersyncers/syncer_test.go +++ b/usersync/usersyncers/syncer_test.go @@ -15,9 +15,11 @@ func TestNewSyncerMap(t *testing.T) { cfg := &config.Configuration{ Adapters: map[string]config.Adapter{ string(openrtb_ext.Bidder33Across): syncConfig, + string(openrtb_ext.BidderAcuityAds): syncConfig, string(openrtb_ext.BidderAdform): syncConfig, string(openrtb_ext.BidderAdkernel): syncConfig, string(openrtb_ext.BidderAdkernelAdn): syncConfig, + string(openrtb_ext.BidderAdman): syncConfig, string(openrtb_ext.BidderAdmixer): syncConfig, string(openrtb_ext.BidderAdOcean): syncConfig, string(openrtb_ext.BidderAdpone): syncConfig, @@ -25,11 +27,14 @@ func TestNewSyncerMap(t *testing.T) { string(openrtb_ext.BidderAdtelligent): syncConfig, string(openrtb_ext.BidderAdvangelists): syncConfig, string(openrtb_ext.BidderAJA): syncConfig, + string(openrtb_ext.BidderAMX): syncConfig, string(openrtb_ext.BidderAppnexus): syncConfig, string(openrtb_ext.BidderAvocet): syncConfig, string(openrtb_ext.BidderBeachfront): syncConfig, string(openrtb_ext.BidderBeintoo): syncConfig, string(openrtb_ext.BidderBrightroll): syncConfig, + string(openrtb_ext.BidderColossus): syncConfig, + string(openrtb_ext.BidderConnectAd): syncConfig, string(openrtb_ext.BidderConsumable): syncConfig, string(openrtb_ext.BidderConversant): syncConfig, string(openrtb_ext.BidderCpmstar): syncConfig, @@ -44,14 +49,18 @@ func TestNewSyncerMap(t *testing.T) { string(openrtb_ext.BidderGrid): syncConfig, string(openrtb_ext.BidderGumGum): syncConfig, string(openrtb_ext.BidderImprovedigital): syncConfig, + string(openrtb_ext.BidderInvibes): syncConfig, string(openrtb_ext.BidderIx): syncConfig, + string(openrtb_ext.BidderKrushmedia): syncConfig, string(openrtb_ext.BidderLifestreet): syncConfig, string(openrtb_ext.BidderLockerDome): syncConfig, + string(openrtb_ext.BidderLogicad): syncConfig, string(openrtb_ext.BidderLunaMedia): syncConfig, string(openrtb_ext.BidderMarsmedia): syncConfig, string(openrtb_ext.BidderMgid): syncConfig, string(openrtb_ext.BidderNanoInteractive): syncConfig, string(openrtb_ext.BidderNinthDecimal): syncConfig, + string(openrtb_ext.BidderNoBid): syncConfig, string(openrtb_ext.BidderOpenx): syncConfig, string(openrtb_ext.BidderPubmatic): syncConfig, string(openrtb_ext.BidderPulsepoint): syncConfig, @@ -62,7 +71,9 @@ func TestNewSyncerMap(t *testing.T) { string(openrtb_ext.BidderSomoaudience): syncConfig, string(openrtb_ext.BidderSonobi): syncConfig, string(openrtb_ext.BidderSovrn): syncConfig, + string(openrtb_ext.BidderSmartadserver): syncConfig, string(openrtb_ext.BidderSmartRTB): syncConfig, + string(openrtb_ext.BidderSmartyAds): syncConfig, string(openrtb_ext.BidderSynacormedia): syncConfig, string(openrtb_ext.BidderTelaria): syncConfig, string(openrtb_ext.BidderTriplelift): syncConfig, @@ -85,14 +96,19 @@ func TestNewSyncerMap(t *testing.T) { openrtb_ext.BidderAdhese: true, openrtb_ext.BidderAdoppler: true, openrtb_ext.BidderApplogy: true, + openrtb_ext.BidderInMobi: true, openrtb_ext.BidderKidoz: true, openrtb_ext.BidderKubient: true, openrtb_ext.BidderMobileFuse: true, openrtb_ext.BidderOrbidder: true, openrtb_ext.BidderPubnative: true, + openrtb_ext.BidderSilverMob: true, + openrtb_ext.BidderSmaato: true, openrtb_ext.BidderSpotX: true, openrtb_ext.BidderTappx: true, openrtb_ext.BidderYeahmobi: true, + openrtb_ext.BidderAdprime: true, + openrtb_ext.BidderBetween: true, } for bidder, config := range cfg.Adapters { diff --git a/util/httputil/httputil.go b/util/httputil/httputil.go new file mode 100644 index 00000000000..93bcca2a8c5 --- /dev/null +++ b/util/httputil/httputil.go @@ -0,0 +1,99 @@ +package httputil + +import ( + "net" + "net/http" + "strings" + + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" +) + +var ( + trueClientIP = http.CanonicalHeaderKey("True-Client-IP") + xForwardedProto = http.CanonicalHeaderKey("X-Forwarded-Proto") + xForwardedFor = http.CanonicalHeaderKey("X-Forwarded-For") + xRealIP = http.CanonicalHeaderKey("X-Real-IP") +) + +const ( + https = "https" +) + +// IsSecure determines if a http request uses https. +func IsSecure(r *http.Request) bool { + if strings.EqualFold(r.Header.Get(xForwardedProto), https) { + return true + } + + if strings.EqualFold(r.URL.Scheme, https) { + return true + } + + if r.TLS != nil { + return true + } + + return false +} + +// FindIP returns the first ip address found in the http request matching the predicate v. +func FindIP(r *http.Request, v iputil.IPValidator) (net.IP, iputil.IPVersion) { + if ip, ver := findTrueClientIP(r, v); ip != nil { + return ip, ver + } + + if ip, ver := findForwardedFor(r, v); ip != nil { + return ip, ver + } + + if ip, ver := findRealIP(r, v); ip != nil { + return ip, ver + } + + if ip, ver := findRemoteAddr(r, v); ip != nil { + return ip, ver + } + + return nil, iputil.IPvUnknown +} + +func findTrueClientIP(r *http.Request, v iputil.IPValidator) (net.IP, iputil.IPVersion) { + if value := r.Header.Get(trueClientIP); value != "" { + value = strings.TrimSpace(value) + if ip, ver := iputil.ParseIP(value); ip != nil && v.IsValid(ip, ver) { + return ip, ver + } + } + return nil, iputil.IPvUnknown +} + +func findForwardedFor(r *http.Request, v iputil.IPValidator) (net.IP, iputil.IPVersion) { + if value := r.Header.Get(xForwardedFor); value != "" { + for _, p := range strings.Split(value, ",") { + p = strings.TrimSpace(p) + if ip, ver := iputil.ParseIP(p); ip != nil && v.IsValid(ip, ver) { + return ip, ver + } + } + } + return nil, iputil.IPvUnknown +} + +func findRealIP(r *http.Request, v iputil.IPValidator) (net.IP, iputil.IPVersion) { + if value := r.Header.Get(xRealIP); value != "" { + value = strings.TrimSpace(value) + if ip, ver := iputil.ParseIP(value); ip != nil && v.IsValid(ip, ver) { + return ip, ver + } + } + return nil, iputil.IPvUnknown +} + +func findRemoteAddr(r *http.Request, v iputil.IPValidator) (net.IP, iputil.IPVersion) { + if host, _, err := net.SplitHostPort(r.RemoteAddr); err == nil { + if ip, ver := iputil.ParseIP(host); ip != nil && v.IsValid(ip, ver) { + return ip, ver + } + } + return nil, iputil.IPvUnknown +} diff --git a/util/httputil/httputil_test.go b/util/httputil/httputil_test.go new file mode 100644 index 00000000000..7b6a9a504f1 --- /dev/null +++ b/util/httputil/httputil_test.go @@ -0,0 +1,327 @@ +package httputil + +import ( + "crypto/tls" + "net" + "net/http" + "testing" + + "github.com/PubMatic-OpenWrap/prebid-server/util/iputil" + "github.com/stretchr/testify/assert" +) + +func TestIsSecure(t *testing.T) { + testCases := []struct { + description string + url string + xForwardedProto string + tls bool + expectIsSecure bool + }{ + { + description: "HTTP", + url: "http://host.com", + expectIsSecure: false, + }, + { + description: "HTTPS - Forwarded Protocol", + url: "http://host.com", + xForwardedProto: "https", + expectIsSecure: true, + }, + { + description: "HTTPS - Forwarded Protocol - Case Insensitive", + url: "http://host.com", + xForwardedProto: "HTTPS", + expectIsSecure: true, + }, + { + description: "HTTPS - Protocol", + url: "https://host.com", + expectIsSecure: true, + }, + { + description: "HTTPS - Protocol - Case Insensitive", + url: "HTTPS://host.com", + expectIsSecure: true, + }, + { + description: "HTTPS - TLS", + url: "http://host.com", + tls: true, + expectIsSecure: true, + }, + } + + for _, test := range testCases { + request, err := http.NewRequest("GET", test.url, nil) + if err != nil { + t.Fatalf("Unable to create test http request. Err: %v", err) + } + if test.xForwardedProto != "" { + request.Header.Add("X-Forwarded-Proto", test.xForwardedProto) + } + if test.tls { + request.TLS = &tls.ConnectionState{} + } + + result := IsSecure(request) + + assert.Equal(t, test.expectIsSecure, result, test.description) + } +} + +func TestFindIP(t *testing.T) { + alwaysTrue := hardcodedResponseIPValidator{response: true} + alwaysFalse := hardcodedResponseIPValidator{response: false} + + testCases := []struct { + description string + trueClientIP string + xForwardedFor string + xRealIP string + remoteAddr string + validator iputil.IPValidator + expectedIP net.IP + expectedVer iputil.IPVersion + }{ + { + description: "No Address", + expectedIP: nil, + expectedVer: iputil.IPvUnknown, + }, + { + description: "False Validator - IPv4", + trueClientIP: "1.1.1.1", + xForwardedFor: "2.2.2.2, 3.3.3.3", + xRealIP: "4.4.4.4", + remoteAddr: "5.5.5.5:5", + validator: alwaysFalse, + expectedIP: nil, + expectedVer: iputil.IPvUnknown, + }, + { + description: "False Validator - IPv6", + trueClientIP: "1111::", + xForwardedFor: "2222::, 3333::", + xRealIP: "4444::", + remoteAddr: "[5555::]:5]", + validator: alwaysFalse, + expectedIP: nil, + expectedVer: iputil.IPvUnknown, + }, + { + description: "True Validator - IPv4 - True Client IP", + trueClientIP: "1.1.1.1", + xForwardedFor: "2.2.2.2, 3.3.3.3", + xRealIP: "4.4.4.4", + remoteAddr: "5.5.5.5:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("1.1.1.1"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv4 - True Client IP - Ignore Whitespace", + trueClientIP: " 1.1.1.1 ", + xForwardedFor: "2.2.2.2, 3.3.3.3", + xRealIP: "4.4.4.4", + remoteAddr: "5.5.5.5:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("1.1.1.1"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv4 - X Forwarded For", + trueClientIP: "", + xForwardedFor: "2.2.2.2, 3.3.3.3", + xRealIP: "4.4.4.4", + remoteAddr: "5.5.5.5:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("2.2.2.2"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv4 - X Forwarded For - Ignore Whitespace", + trueClientIP: "", + xForwardedFor: " 2.2.2.2, 3.3.3.3 ", + xRealIP: "4.4.4.4", + remoteAddr: "5.5.5.5:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("2.2.2.2"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv4 - X Real IP", + trueClientIP: "", + xForwardedFor: "", + xRealIP: "4.4.4.4", + remoteAddr: "5.5.5.5:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("4.4.4.4"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv4 - X Real IP - Ignore Whitespace", + trueClientIP: "", + xForwardedFor: "", + xRealIP: " 4.4.4.4 ", + remoteAddr: "5.5.5.5:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("4.4.4.4"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv4 - Remote Address", + trueClientIP: "", + xForwardedFor: "", + xRealIP: "", + remoteAddr: "5.5.5.5:80", + validator: alwaysTrue, + expectedIP: net.ParseIP("5.5.5.5"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - IPv6 - True Client IP", + trueClientIP: "1111::", + xForwardedFor: "2222::, 3333::", + xRealIP: "4444::", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("1111::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - IPv6 - True Client IP - Ignore Whitespace", + trueClientIP: " 1111:: ", + xForwardedFor: "2222::, 3333::", + xRealIP: "4444::", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("1111::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - IPv6 - X Forwarded For", + trueClientIP: "", + xForwardedFor: "2222::, 3333::", + xRealIP: "4444::", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("2222::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - IPv6 - X Forwarded For - Ignore Whitespace", + trueClientIP: "", + xForwardedFor: " 2222::, 3333:: ", + xRealIP: "4444::", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("2222::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - IPv6 - X Real IP", + trueClientIP: "", + xForwardedFor: "", + xRealIP: "4444::", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("4444::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - IPv6 - X Real IP - Ignore Whitespace", + trueClientIP: "", + xForwardedFor: "", + xRealIP: " 4444:: ", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("4444::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - IPv6 - Remote Address", + trueClientIP: "", + xForwardedFor: "", + xRealIP: "", + remoteAddr: "[5555::]:5", + validator: alwaysTrue, + expectedIP: net.ParseIP("5555::"), + expectedVer: iputil.IPv6, + }, + { + description: "True Validator - Malformed - All", + trueClientIP: "malformed", + xForwardedFor: "malformed", + xRealIP: "malformed", + remoteAddr: "malformed", + validator: alwaysTrue, + expectedIP: nil, + expectedVer: iputil.IPvUnknown, + }, + { + description: "True Validator - Malformed - Some", + trueClientIP: "malformed", + xForwardedFor: "malformed", + xRealIP: "4.4.4.4", + remoteAddr: "malformed", + validator: alwaysTrue, + expectedIP: net.ParseIP("4.4.4.4"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - Malformed - X Forwarded For - IPv4", + trueClientIP: "malformed", + xForwardedFor: "malformed, 4.4.4.4, 3333::, malformed", + xRealIP: "malformed", + remoteAddr: "malformed", + validator: alwaysTrue, + expectedIP: net.ParseIP("4.4.4.4"), + expectedVer: iputil.IPv4, + }, + { + description: "True Validator - Malformed - X Forwarded For - IPv6", + trueClientIP: "malformed", + xForwardedFor: "malformed, 3333::, 4.4.4.4, malformed", + xRealIP: "malformed", + remoteAddr: "malformed", + validator: alwaysTrue, + expectedIP: net.ParseIP("3333::"), + expectedVer: iputil.IPv6, + }, + } + + for _, test := range testCases { + // Build Request + request, err := http.NewRequest("GET", "http://anyurl.com", nil) + if err != nil { + t.Fatalf("Unable to create test http request. Err: %v", err) + } + if test.trueClientIP != "" { + request.Header.Add("True-Client-IP", test.trueClientIP) + } + if test.xForwardedFor != "" { + request.Header.Add("X-Forwarded-For", test.xForwardedFor) + } + if test.xRealIP != "" { + request.Header.Add("X-Real-IP", test.xRealIP) + } + request.RemoteAddr = test.remoteAddr + + // Run Test + ip, ver := FindIP(request, test.validator) + + // Assertions + assert.Equal(t, test.expectedIP, ip, test.description+":ip") + assert.Equal(t, test.expectedVer, ver, test.description+":ver") + } +} + +type hardcodedResponseIPValidator struct { + response bool +} + +func (v hardcodedResponseIPValidator) IsValid(net.IP, iputil.IPVersion) bool { + return v.response +} diff --git a/util/iputil/parse.go b/util/iputil/parse.go new file mode 100644 index 00000000000..bcb00760e22 --- /dev/null +++ b/util/iputil/parse.go @@ -0,0 +1,27 @@ +package iputil + +import ( + "net" + "strings" +) + +// IPVersion is the numerical version of the IP address spec (4 or 6). +type IPVersion int + +// IP address versions. +const ( + IPvUnknown IPVersion = 0 + IPv4 IPVersion = 4 + IPv6 IPVersion = 6 +) + +// ParseIP parses v as an ip address returning the result and version, or nil and unknown if invalid. +func ParseIP(v string) (net.IP, IPVersion) { + if ip := net.ParseIP(v); ip != nil { + if strings.ContainsRune(v, ':') { + return ip, IPv6 + } + return ip, IPv4 + } + return nil, IPvUnknown +} diff --git a/util/iputil/parse_test.go b/util/iputil/parse_test.go new file mode 100644 index 00000000000..53431b0f2a9 --- /dev/null +++ b/util/iputil/parse_test.go @@ -0,0 +1,30 @@ +package iputil + +import ( + "net" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestParseIP(t *testing.T) { + testCases := []struct { + input string + expectedVer IPVersion + expectedIP net.IP + }{ + {"", IPvUnknown, nil}, + {"1.1.1.1", IPv4, net.IPv4(1, 1, 1, 1)}, + {"-1.-1.-1.-1", IPvUnknown, nil}, + {"256.256.256.256", IPvUnknown, nil}, + {"::ffff:1.1.1.1", IPv6, net.IP{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 1, 1, 1, 1}}, + {"0101::", IPv6, net.IP{1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}}, + {"zzzz::", IPvUnknown, nil}, + } + + for _, test := range testCases { + ip, ver := ParseIP(test.input) + assert.Equal(t, test.expectedVer, ver) + assert.Equal(t, test.expectedIP, ip) + } +} diff --git a/util/iputil/validator.go b/util/iputil/validator.go new file mode 100644 index 00000000000..e4b822f0c7c --- /dev/null +++ b/util/iputil/validator.go @@ -0,0 +1,48 @@ +package iputil + +import ( + "net" +) + +// IPValidator is the interface for validating an ip address and version. +type IPValidator interface { + // IsValid returns true when an IP address is determined to be valid. + IsValid(net.IP, IPVersion) bool +} + +// PublicNetworkIPValidator validates an ip address which is not contained in the list of known private networks. +type PublicNetworkIPValidator struct { + IPv4PrivateNetworks []net.IPNet + IPv6PrivateNetworks []net.IPNet +} + +// IsValid implements the IPValidator interface. +func (v PublicNetworkIPValidator) IsValid(ip net.IP, ver IPVersion) bool { + var privateNetworks []net.IPNet + switch ver { + case IPv4: + privateNetworks = v.IPv4PrivateNetworks + case IPv6: + privateNetworks = v.IPv6PrivateNetworks + default: + return false + } + + for _, ipNet := range privateNetworks { + if ipNet.Contains(ip) { + return false + } + } + + return true +} + +// VersionIPValidator validates an ip address based on the desired ip version. +type VersionIPValidator struct { + Version IPVersion +} + +// IsValid implements the IPValidator interface. +func (v VersionIPValidator) IsValid(ip net.IP, ver IPVersion) bool { + return ver == v.Version +} diff --git a/util/iputil/validator_test.go b/util/iputil/validator_test.go new file mode 100644 index 00000000000..4419af22c04 --- /dev/null +++ b/util/iputil/validator_test.go @@ -0,0 +1,222 @@ +package iputil + +import ( + "net" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestPublicNetworkIPValidator(t *testing.T) { + ipv4Network1 := net.IPNet{IP: net.ParseIP("1.0.0.0"), Mask: net.CIDRMask(8, 32)} + ipv4Network2 := net.IPNet{IP: net.ParseIP("2.0.0.0"), Mask: net.CIDRMask(8, 32)} + + ipv6Network1 := net.IPNet{IP: net.ParseIP("3300::"), Mask: net.CIDRMask(8, 128)} + ipv6Network2 := net.IPNet{IP: net.ParseIP("4400::"), Mask: net.CIDRMask(8, 128)} + + testCases := []struct { + description string + ip net.IP + ver IPVersion + ipv4PrivateNetworks []net.IPNet + ipv6PrivateNetworks []net.IPNet + expected bool + }{ + { + description: "IPv4 - Public - None", + ip: net.ParseIP("1.1.1.1"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{}, + ipv6PrivateNetworks: []net.IPNet{}, + expected: true, + }, + { + description: "IPv4 - Public - One", + ip: net.ParseIP("2.2.2.2"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{}, + expected: true, + }, + { + description: "IPv4 - Public - Many", + ip: net.ParseIP("3.3.3.3"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network2}, + ipv6PrivateNetworks: []net.IPNet{}, + expected: true, + }, + { + description: "IPv4 - Private - One", + ip: net.ParseIP("1.1.1.1"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{}, + expected: false, + }, + { + description: "IPv4 - Private - Many", + ip: net.ParseIP("2.2.2.2"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network2}, + ipv6PrivateNetworks: []net.IPNet{}, + expected: false, + }, + { + description: "IPv6 - Public - None", + ip: net.ParseIP("3333::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{}, + ipv6PrivateNetworks: []net.IPNet{}, + expected: true, + }, + { + description: "IPv6 - Public - One", + ip: net.ParseIP("4444::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1}, + expected: true, + }, + { + description: "IPv6 - Public - Many", + ip: net.ParseIP("5555::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: true, + }, + { + description: "IPv6 - Private - One", + ip: net.ParseIP("3333::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1}, + expected: false, + }, + { + description: "IPv6 - Private - Many", + ip: net.ParseIP("4444::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: false, + }, + { + description: "Mixed - Unknown", + ip: net.ParseIP("3.3.3.3"), + ver: IPvUnknown, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: false, + }, + { + description: "Mixed - Public - IPv4", + ip: net.ParseIP("3.3.3.3"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: true, + }, + { + description: "Mixed - Public - IPv6", + ip: net.ParseIP("5555::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: true, + }, + { + description: "Mixed - Private - IPv4", + ip: net.ParseIP("1.1.1.1"), + ver: IPv4, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: false, + }, + { + description: "Mixed - Private - IPv6", + ip: net.ParseIP("3333::"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{ipv4Network1, ipv4Network1}, + ipv6PrivateNetworks: []net.IPNet{ipv6Network1, ipv6Network2}, + expected: false, + }, + { + description: "Mixed - Public - IPv6 Encoded IPv4", + ip: net.ParseIP("::FFFF:1.1.1.1"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{{IP: net.ParseIP("1.0.0.0"), Mask: net.CIDRMask(8, 32)}}, + ipv6PrivateNetworks: []net.IPNet{{IP: net.ParseIP("::FFFF:2.0.0.0"), Mask: net.CIDRMask(108, 128)}}, + expected: true, + }, + { + description: "Mixed - Private - IPv6 Encoded IPv4", + ip: net.ParseIP("::FFFF:2.2.2.2"), + ver: IPv6, + ipv4PrivateNetworks: []net.IPNet{{IP: net.ParseIP("1.0.0.0"), Mask: net.CIDRMask(8, 32)}}, + ipv6PrivateNetworks: []net.IPNet{{IP: net.ParseIP("::FFFF:2.0.0.0"), Mask: net.CIDRMask(108, 128)}}, + expected: false, + }, + } + + for _, test := range testCases { + requestValidation := PublicNetworkIPValidator{ + IPv4PrivateNetworks: test.ipv4PrivateNetworks, + IPv6PrivateNetworks: test.ipv6PrivateNetworks, + } + + result := requestValidation.IsValid(test.ip, test.ver) + + assert.Equal(t, test.expected, result, test.description) + } +} + +func TestVersionIPValidator(t *testing.T) { + testCases := []struct { + description string + validatorVersion IPVersion + ip net.IP + ipVer IPVersion + expected bool + }{ + { + description: "IPv4", + validatorVersion: IPv4, + ip: net.ParseIP("1.1.1.1"), + ipVer: IPv4, + expected: true, + }, + { + description: "IPv4 - Given Unknown", + validatorVersion: IPv4, + ip: nil, + ipVer: IPvUnknown, + expected: false, + }, + { + description: "IPv6", + validatorVersion: IPv6, + ip: net.ParseIP("1111::"), + ipVer: IPv6, + expected: true, + }, + { + description: "IPv6 - Given Unknown", + validatorVersion: IPv6, + ip: nil, + ipVer: IPvUnknown, + expected: false, + }, + } + + for _, test := range testCases { + m := VersionIPValidator{ + Version: test.validatorVersion, + } + + result := m.IsValid(test.ip, test.ipVer) + + assert.Equal(t, test.expected, result) + } +} diff --git a/util/maputil/maputil.go b/util/maputil/maputil.go new file mode 100644 index 00000000000..0d1d7dbb51c --- /dev/null +++ b/util/maputil/maputil.go @@ -0,0 +1,21 @@ +package maputil + +// ReadEmbeddedMap reads element k from the map m as a map[string]interface{}. +func ReadEmbeddedMap(m map[string]interface{}, k string) (map[string]interface{}, bool) { + if v, ok := m[k]; ok { + vCasted, ok := v.(map[string]interface{}) + return vCasted, ok + } + + return nil, false +} + +// ReadEmbeddedSlice reads element k from the map m as a []interface{}. +func ReadEmbeddedSlice(m map[string]interface{}, k string) ([]interface{}, bool) { + if v, ok := m[k]; ok { + vCasted, ok := v.([]interface{}) + return vCasted, ok + } + + return nil, false +} diff --git a/util/maputil/maputil_test.go b/util/maputil/maputil_test.go new file mode 100644 index 00000000000..2e6955cec9b --- /dev/null +++ b/util/maputil/maputil_test.go @@ -0,0 +1,113 @@ +package maputil + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestReadEmbeddedMap(t *testing.T) { + testCases := []struct { + description string + value map[string]interface{} + key string + expectedMap map[string]interface{} + expectedOK bool + }{ + { + description: "Nil", + value: nil, + key: "", + expectedMap: nil, + expectedOK: false, + }, + { + description: "Empty", + value: map[string]interface{}{}, + key: "foo", + expectedMap: nil, + expectedOK: false, + }, + { + description: "Success", + value: map[string]interface{}{"foo": map[string]interface{}{"bar": 42}}, + key: "foo", + expectedMap: map[string]interface{}{"bar": 42}, + expectedOK: true, + }, + { + description: "Not Found", + value: map[string]interface{}{"foo": map[string]interface{}{"bar": 42}}, + key: "notFound", + expectedMap: nil, + expectedOK: false, + }, + { + description: "Wrong Type", + value: map[string]interface{}{"foo": 42}, + key: "foo", + expectedMap: nil, + expectedOK: false, + }, + } + + for _, test := range testCases { + resultMap, resultOK := ReadEmbeddedMap(test.value, test.key) + + assert.Equal(t, test.expectedMap, resultMap, test.description+":map") + assert.Equal(t, test.expectedOK, resultOK, test.description+":ok") + } +} + +func TestReadEmbeddedSlice(t *testing.T) { + testCases := []struct { + description string + value map[string]interface{} + key string + expectedSlice []interface{} + expectedOK bool + }{ + { + description: "Nil", + value: nil, + key: "", + expectedSlice: nil, + expectedOK: false, + }, + { + description: "Empty", + value: map[string]interface{}{}, + key: "foo", + expectedSlice: nil, + expectedOK: false, + }, + { + description: "Success", + value: map[string]interface{}{"foo": []interface{}{42}}, + key: "foo", + expectedSlice: []interface{}{42}, + expectedOK: true, + }, + { + description: "Not Found", + value: map[string]interface{}{"foo": []interface{}{42}}, + key: "notFound", + expectedSlice: nil, + expectedOK: false, + }, + { + description: "Wrong Type", + value: map[string]interface{}{"foo": 42}, + key: "foo", + expectedSlice: nil, + expectedOK: false, + }, + } + + for _, test := range testCases { + resultSlice, resultOK := ReadEmbeddedSlice(test.value, test.key) + + assert.Equal(t, test.expectedSlice, resultSlice, test.description+":slicd") + assert.Equal(t, test.expectedOK, resultOK, test.description+":ok") + } +} diff --git a/util/task/ticker_task.go b/util/task/ticker_task.go new file mode 100644 index 00000000000..a8d523b75d5 --- /dev/null +++ b/util/task/ticker_task.go @@ -0,0 +1,53 @@ +package task + +import ( + "time" +) + +type Runner interface { + Run() error +} + +type TickerTask struct { + interval time.Duration + runner Runner + done chan struct{} +} + +func NewTickerTask(interval time.Duration, runner Runner) *TickerTask { + return &TickerTask{ + interval: interval, + runner: runner, + done: make(chan struct{}), + } +} + +// Start runs the task immediately and then schedules the task to run periodically +// if a positive fetching interval has been specified. +func (t *TickerTask) Start() { + t.runner.Run() + + if t.interval > 0 { + go t.runRecurring() + } +} + +// Stop stops the periodic task but the task runner maintains state +func (t *TickerTask) Stop() { + close(t.done) +} + +// run creates a ticker that ticks at the specified interval. On each tick, +// the task is executed +func (t *TickerTask) runRecurring() { + ticker := time.NewTicker(t.interval) + + for { + select { + case <-ticker.C: + t.runner.Run() + case <-t.done: + return + } + } +} diff --git a/util/task/ticker_task_test.go b/util/task/ticker_task_test.go new file mode 100644 index 00000000000..92cf6835ea6 --- /dev/null +++ b/util/task/ticker_task_test.go @@ -0,0 +1,63 @@ +package task_test + +import ( + "testing" + "time" + + "github.com/PubMatic-OpenWrap/prebid-server/util/task" + "github.com/stretchr/testify/assert" +) + +type MockRunner struct { + RunCount int +} + +func (mcc *MockRunner) Run() error { + mcc.RunCount++ + return nil +} + +func TestStartWithSingleRun(t *testing.T) { + // Setup: + runner := &MockRunner{RunCount: 0} + interval := 0 * time.Millisecond + ticker := task.NewTickerTask(interval, runner) + + // Execute: + ticker.Start() + time.Sleep(10 * time.Millisecond) + + // Verify: + assert.Equal(t, runner.RunCount, 1, "runner should have run one time") +} + +func TestStartWithPeriodicRun(t *testing.T) { + // Setup: + runner := &MockRunner{RunCount: 0} + interval := 10 * time.Millisecond + ticker := task.NewTickerTask(interval, runner) + + // Execute: + ticker.Start() + time.Sleep(25 * time.Millisecond) + ticker.Stop() + + // Verify: + assert.Equal(t, runner.RunCount, 3, "runner should have run three times") +} + +func TestStop(t *testing.T) { + // Setup: + runner := &MockRunner{RunCount: 0} + interval := 10 * time.Millisecond + ticker := task.NewTickerTask(interval, runner) + + // Execute: + ticker.Start() + time.Sleep(25 * time.Millisecond) + ticker.Stop() + time.Sleep(25 * time.Millisecond) // wait in case stop failed so additional runs can happen + + // Verify: + assert.Equal(t, runner.RunCount, 3, "runner should have run three times") +} diff --git a/util/timeutil/time.go b/util/timeutil/time.go new file mode 100644 index 00000000000..e8eaae7d61f --- /dev/null +++ b/util/timeutil/time.go @@ -0,0 +1,16 @@ +package timeutil + +import ( + "time" +) + +type Time interface { + Now() time.Time +} + +// RealTime wraps the time package for testability +type RealTime struct{} + +func (c *RealTime) Now() time.Time { + return time.Now() +}