Psiphon travis build pipeline #40
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s (unexplainably) missing from the latest psiandroid image build
…o run a full build again.
jmwample
added a commit
that referenced
this pull request
May 4, 2020
jmwample
added a commit
that referenced
this pull request
May 4, 2020
jmwample
added a commit
that referenced
this pull request
May 11, 2020
* initial test incorporating things from dark-decoy build process * Disabled unreliable test * disabled another unreliable test * Attempting to checkout specific commit for build process * use mercurial checkout * adding env vars for Psiphon repo commits used * minor fixes - relying on travis ci online env vars over local * accomodating references to sergeyfrolov/gotapdance in psiphon for build process * fixes docker image based build issues as in PR #40 * fixes docker image based build issues as in PR #40 (missed a save) * fixing issue #38 for msater so builds will work. * using old version (1.12.x) of golang for build, psiphon only claims to work with 1.13+ * Testing a theory that a change in the conjure embeddedvalues needs to be reflected in master
jmwample
added a commit
that referenced
this pull request
Aug 28, 2020
* initial test incorporating things from dark-decoy build process * Disabled unreliable test * disabled another unreliable test * Attempting to checkout specific commit for build process * use mercurial checkout * adding env vars for Psiphon repo commits used * minor fixes - relying on travis ci online env vars over local * accomodating references to sergeyfrolov/gotapdance in psiphon for build process * fixes docker image based build issues as in PR #40 * fixes docker image based build issues as in PR #40 (missed a save) * fixing issue #38 for msater so builds will work. * using old version (1.12.x) of golang for build, psiphon only claims to work with 1.13+ * Testing a theory that a change in the conjure embeddedvalues needs to be reflected in master * build updates for master, also ahead of master for some reason * updating embedded list to solve issue #32
jmwample
added a commit
that referenced
this pull request
Oct 5, 2020
* WIP: implement client for dark decoy * WIP: wippy wip mcwipface * WIP: first working client prototype * WIP: make transparent proxy listen forever+gofmt * WIP: use obfs4 instead of transparent proxy * gofmt * Conjure: integrated prototype with 2-way-proxy * Allow Client to drive V4/V6 decoy ip selection * pivot to bool for ipv6 support over uint32 - passing generation tests * Implemented v6 support check and fallback * updated naming and CLI usage compiles and registers with dark decoy station * v4v6 fallback with proxy header support * added command line options for specifying proxy header use and Ipv6 inclusion/exclusion * using reserved IP blocks to identify ipv6support instead of interface naming conventions * conjure: finalize move to the new repo (#12) This change replaces all old links to sergeyfrolov/gotapdance with refraction-networking/gotapdance * fixes leading to merge into dark-decoy * Travis build (#17) * added secret value and removed tests which run forever. testing travis pipeline * adding forgotten files to commit * commented out failing test * linter fixes * Trying 18.04 bionic for build with newer openssl * possible issue fix * fixing patch file for conjure build * fixing tapdance build * Clean Disable for tests * completing requested changes * ClientConf digest during build (#18) * added secret value and removed tests which run forever. testing travis pipeline * adding forgotten files to commit * commented out failing test * linter fixes * Trying 18.04 bionic for build with newer openssl * possible issue fix * fixing patch file for conjure build * fixing tapdance build * Clean Disable for tests * digest ClientConf of current branch into Psiphon Android App. * Added Psiphon Specific comments * Small fixes * TapDance -> Refraction * typo * Travis build fixes (#23) * added secret value and removed tests which run forever. testing travis pipeline * adding forgotten files to commit * commented out failing test * linter fixes * Trying 18.04 bionic for build with newer openssl * possible issue fix * fixing patch file for conjure build * fixing tapdance build * Clean Disable for tests * digest ClientConf of current branch into Psiphon Android App. * Added Psiphon Specific comments * Small fixes * Adding Specific commit checkout for building psiphon library and apk * trying older commit * possible fix for build problems based on govendor issues. * accomodating relics of sergeyfrolov/gotapdance repo in Psiphon * testing newer psiphon system commit * Split Registration and Connection -- no obvious errors, completely untested. * adding test functions to make sure conjure functionality is reproducible. -- not fully working * cleaning registration context, and callback * Added V6 addresses to decoys in ClientConf * added script * quick cleanup * success reading v6decoys from clientConf * IPv6 Decoy deduplication * Bug Hunting * v6 fixes and patches, bug hunting * adding broader v4v6 support * Bug Hunting * cleaning tests * fix inneffectial assign error in gometalinter * adding v6 support to assets to track across sessions * registration fixes * quick fixes * adding test v6 decoy that definitely goes past curveball * support patches * fixing bugs with min transport -- DATA WORKING * test fix for build system * gometalinter fixes * v6 cleanup (no longer caching v6 support in assets * Remove dependence on station_pubkey file in assets folder. Reads default pubkey from ClientConf or locally if ClienConf is unavailable, also change default pubkey to new conjure key * Modify travis build to include Conjure ConsoleClient * format fix * mistaken about config * updating to latest psiphon core commit * travis restructure to try to fix broken build * adding new fields required byPsiphon in EmbeddedValues * bypassing old docker image for android build * building on updated docker image * gradlew build issue with min api version * enbale debug * adding patch for psiphon compatibility with java 1.8 * trying to fix regions redirect issues * Conjure MinTransport (#26) * Add Transport to VSP * actually return the correct transport type * Register both (#31) * Using Dev Client Conf (dev decoys) on full conjure build * bypass decoy check * changed default pubkey in dev ClientConf * full logging enable * allow client to request registration of both v4 and v6 at the same time * gometalinter fixes * updating protobuf to match for client and station * small cleaning * patched tcpdialer specified in dialer into conjure registration and connection * adding logging update to trigger rebuild * use TcpDialer for connect * add build apk script * Conjure full (#35) * using ALL decoys instead of just dev. Will likely fail to register for multiple sessions in a row. * added hard coded decoy that should consistently work, working from cli * Td cj compat (#34) * Added second key to clientconf for conjure to use specially -- tested, working * disabled automatic clientconf updates. untested * cli compatible to run either conjure or tapdance * added two keys (production pubkey for tapdance and dev pubkey for conjure) to ClientConf * added subnet usable by windy egret from Internet2 (CU Bouler AS104) or otherwise. * subnets changed so phantoms selected in tests changed * rolled back changed to tapdance -- compiling connecting working for cli * updated subnets one last time. * updated subnets, had to update tests * removing deadcode created by tapdance rollback * re-implement phantom selection as it's own submodule. (#36) * re-implement phantom selection as it's own submodule with testing and support for weighted subnet lists. * comments and light cleanup * Update/clientconf (#37) * update clientConf to use decoys from generation 957 instead of 538 which it was on. v4 decoys ~1580 -> ~1250 and v6 decoys ~40 -> ~30 * real update (forgot to commit after latest changes * addressing issue #38 by moving ed25519 code locally to keep elligator working as the original repo is no longer suppoting it. * corrected github links that I missed * missed one * fix sed issue in build * applying patch for bit leak from pull request * TestObfuscationRandomness test was not passing with patch applied * attempted hot patch for missing go dependencies in build pipeline * simple elligator util * fix -b representation issue * Debug travis docker golang problems * cleanup tools for build reasons * bit always zero in test -> expected behavior for now * added debug logging reasons for failed registration connections * Update/local ed25519 (#39) * addressing issue #38 by moving ed25519 code locally to keep elligator working as the original repo is no longer suppoting it. * corrected github links that I missed * missed one * fix sed issue in build * applying patch for bit leak from pull request * TestObfuscationRandomness test was not passing with patch applied * attempted hot patch for missing go dependencies in build pipeline * simple elligator util * fix -b representation issue * Debug travis docker golang problems * cleanup tools for build reasons * bit always zero in test -> expected behavior for now * added debug logging reasons for failed registration connections Co-authored-by: Eric Wustrow <ewust@colorado.edu> * randomizing two bits on the client see PR #39 and issue #38 * Psiphon travis build pipeline (#40) * trying build with latest releases of psiphon code * build breaking on docker build * build breaking because of go libs available during build * build breaking because of gomobile is failing * stupid mistake * back to basics * packages not pulled * should new be getting utls dependencies * maybe I was overwriting the gomobile in the psiabdroid image * missing utls dependecies in refraction/psiandroid * go get all refraction-networking dependencies inside of the psiandroid docker image * removed dependence on /android-sdk-linux/tools/bin/sdkmanager which is (unexplainably) missing from the latest psiandroid image build * accepting licenses because sdkmanager IS GONE FOR SOME REASON * Still missing some licenses for unknown reason * accepting license that makes it work locally? * added comments with warnings and uncommented all other build stages to run a full build again. * missed one line * updated to clientconf decoy list generation 1119 * added script for testing decoy connecitons * fixed issue with wait group * Always unlock assets when setting * Registration API Functionality (#43) * Basic registration API start * Send shared secret, FSP, VSP in payload * Simplify client-to-API payload * Don't copy protobuf structs * Add support for retries, fallback to API registration * Change APIRegistrar HTTP.Client to pointer * Use API registration in Psiphon app * Fix typo in Psiphon patch * Add test for APIRegistrar * Bump connection delay to 750ms * Make sure to close response body, even when empty * Increase CLI connection delay to 750ms * Fix endpoint in Psiphon patch * Use specified TCPDialer for registration * Move flags from FSP to C2S * Build updates (#47) * test build with alternative psiphon tunnel core branch * issue 1 - the way the branch was pulled * moving to github for psiphon android build * add email to myself for failing builds in major branches * config parsing correction * correct config error * testing solution to issue #32 * revert testing change involving partner library git tag/version * obfs4 Support (#44) * Basic registration API start * Send shared secret, FSP, VSP in payload * Simplify client-to-API payload * Don't copy protobuf structs * Add support for retries, fallback to API registration * Change APIRegistrar HTTP.Client to pointer * Use API registration in Psiphon app * Fix typo in Psiphon patch * Add test for APIRegistrar * Bump connection delay to 750ms * Make sure to close response body, even when empty * Increase CLI connection delay to 750ms * Fix endpoint in Psiphon patch * Use specified TCPDialer for registration * Add obfs4 support * Move flags from FSP to C2S * Remove stutter in transport names * Fix support for NullTransport * Default to MinTransport * Correct test to specify transport * Add registration source field to ZMQPayload (#46) * Add registration source field to ZMQPayload * resolving differences in generated protobuf go source file * updating signaling protobuf golang library Co-authored-by: jmwample <jack.wampler@colorado.edu> Co-authored-by: Jack Wampler <jmwample@users.noreply.github.com> * missed integration items * more fixes * missed a func * Android build tools update for travis-ci * Conjure Build issues and merge compatibility (#51) * build issues * trying build with update all SDK packages * trying build with update wildcard filtered SDK packages * seeing if the new lts psiphon core fixes build issues * missed some docker tags * trying more filtering * trying more filtering * naming test conjure apk Co-authored-by: Sergey Frolov <sergey.frolov@colorado.edu> Co-authored-by: Eric Wustrow <ewust@colorado.edu> Co-authored-by: jmwample <jackjack108@gmail.com> Co-authored-by: Carson Hoffman <c@rsonhoffman.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The build pipeline has been broken for a while now. I have gone back and identified the key issues and corrected them. I am going to walk through the two main issues here.
The
.travis.ymlfile that governs our build relies on docker to run an image specified bya dockerfile provided by psiphon. This dockerfile sets up the environment for building the
psiphon-core-libraryand installs android sdk to be able to build an android APK.Problem 1 -- Pinned gomobile
In the environment setup done by the dockerfile a pinned version of the
gomobilelibrary is included as the latest version at golang.org/x/mobile/cmd/gomobile breaks the build process.At the same time, when we go to build the development APK many of the go libraries used for
gotapdanceandutlsare missing.even though gotapdance is vendored in the psiphon library build directory we cannot just do a
go get ./...as this pulls the latest version ofgomobileas well. Instead we have to cd to the directories of each refraction library we are going to use and pull the dependencies ourselves. Since none of the refraction projects (currently) depend ongomobilethe latest version will not be pulled but all required libraries will be.Problem 2 -- Android SDK licenses
For some reason the version of the android sdk that is installed in the docker image built from the psiphon specification does not have certain licenses accepted. This breaks the build as android refuses to proceed.
Previously we included the command
yes | sdkmanager --updatehowever there is now NO sdkmanager program in the tools for this version of android sdk. I don't really understand this as psiphon doesn't appear to have changed the sdk version included in the docker image since our builds we last working. This means that for some reason the sdk must have removed the sdkmanager from the tools included in this version, otherwise I am missing something.Either way, the work-around is to use a command similar to one in the dockerfile where they attempt to accept the licenses and add it to the front of our run command when we build the APK.
Note: we cannot use
yes | (progam to accept license)as yes sendsycharacters too quickly and will end up declining the licenses.