# TLS-Client ## Preface This TLS Client is built upon and as well as . Big thanks to all contributors so far. Sadly it seems that the original repositories are not maintained anymore and the original \`utls\` repository is not enough without custom modifications. ## What is TLS Fingerprinting? Some people think it is enough to change the user-agent header of a request to let the server think that the client requesting a resource is a specific browser. Nowadays this is not enough, because the server might use a technique to detect the client browser which is called TLS Fingerprinting. Even tho this article is about TLS Fingerprinting in NodeJS it well describes the technique in detail:\ ## Why is this library needed? With this library you are able to create a http client implementing an interface which is similar to golangs net/http client interface. This TLS Client allows you to specify the client profile (Browser) you want to use, when requesting a server. The interface of the HTTP Client looks like the following and extends the base net/http Client interface by some useful functions. Most likely you will use the `Do()` function like you did before with golangs net/http Client. ```go type HttpClient interface { GetCookies(u *url.URL) []*http.Cookie SetCookies(u *url.URL, cookies []*http.Cookie) SetCookieJar(jar http.CookieJar) GetCookieJar() http.CookieJar SetProxy(proxyUrl string) error GetProxy() string SetFollowRedirect(followRedirect bool) GetFollowRedirect() bool CloseIdleConnections() Do(req *http.Request) (*http.Response, error) Get(url string) (resp *http.Response, err error) Head(url string) (resp *http.Response, err error) Post(url, contentType string, body io.Reader) (resp *http.Response, err error) GetBandwidthTracker() bandwidth.BandwidthTracker GetDialer() proxy.ContextDialer GetTLSDialer() TLSDialerFunc } ``` The methods should be more or less self explanatory. # Supported and tested Client Profiles ### Internal Client Profiles The internal client profiles were created and tested by myself. Of course that does not mean they are 100% correct. But i tested them more precise then the contributed profiles. 1. Chrome * 103 (chrome\_103) * 104 (chrome\_104) * 105 (chrome\_105) * 106 (chrome\_106) * 107 (chrome\_107) * 108 (chrome\_108) * 109 (chrome\_109) * 110 (chrome\_110) * 111 (chrome\_111) * 112 (chrome\_112) * 116 with PSK (chrome\_116\_PSK) * 116 with PSK and PQ (chrome\_116\_PSK\_PQ) * 117 (chrome\_117) * 120 (chrome\_120) * 124 (chrome\_124) * 133 (chrome\_133) * 133 with PSK (chrome\_133\_PSK) * 144 (chrome\_144) * 144 with PSK (chrome\_144\_PSK) * 146 (chrome\_146) * 146 with PSK (chrome\_146\_PSK) 2. Brave * 146 (brave\_146) * 146 with PSK (brave\_146\_PSK) 3. Safari * 15.6.1 (safari\_15\_6\_1) * 16.0 (safari\_16\_0) 4. iOS (Safari) * 15.5 (safari\_ios\_15\_5) * 15.6 (safari\_ios\_15\_6) * 16.0 (safari\_ios\_16\_0) * 17.0 (safari\_ios\_17\_0) * 18.0 (safari\_ios\_18\_0) * 18.5 (safari\_ios\_18\_5) * 26.0 (safari\_ios\_26\_0) 5. iPadOS (Safari) * 15.6 (safari\_ipad\_15\_6) 6. Firefox * 102 (firefox\_102) * 104 (firefox\_104) * 105 (firefox\_105) * 106 (firefox\_106) * 108 (firefox\_108) * 110 (firefox\_110) * 117 (firefox\_117) 7. Opera * 89 (opera\_89) * 90 (opera\_90) * 91 (opera\_91) 8. Custom Clients * Zalando iOS Mobile (zalando\_ios\_mobile) * Nike IOS Mobile (nike\_ios\_mobile) * Cloudscraper * MMS IOS (mms\_ios or mms\_ios\_1) * MMS IOS 2 (mms\_ios\_2) * MMS IOS 3 (mms\_ios\_3) * Mesh IOS (mesh\_ios or mesh\_ios\_1) * Confirmed IOS (confirmed\_ios) ### Contributed Client Profiles The contributed client profiles are contributed by people from the community. They are not precisely tested like the internal ones. Please be aware that they might have issues. 1. Chrome * 130 with PSK (chrome\_130\_PSK) * 131 (chrome\_131) * 131 with PSK (chrome\_131\_PSK) 2. OkHttp4 * Android 7 (okhttp4\_android\_7) * Android 8 (okhttp4\_android\_8) * Android 9 (okhttp4\_android\_9) * Android 10 (okhttp4\_android\_10) * Android 11 (okhttp4\_android\_11) * Android 12 (okhttp4\_android\_12) * Android 13 (okhttp4\_android\_13) 3. Custom Clients * Zalando Android Mobile (zalando\_android\_mobile) * Nike Android Mobile (nike\_android\_mobile) * Mesh IOS 2 (mesh\_ios\_2) * Mesh Android (mesh\_android or mesh\_android\_1) * Mesh Android 2 (mesh\_android\_2) * Confirmed Android (confirmed\_android) 4. Firefox * 120 (firefox\_120) * 123 (firefox\_123) * 132 (firefox\_132) * 133 (firefox\_133) * 135 (firefox\_135) * 146 with PSK (firefox\_146\_PSK) * 147 (firefox\_147) * 147 with PSK (firefox\_147\_PSK) * 148 (firefox\_148) You can also provide your own client. See the examples how to do that. All Clients support Random TLS Extension Order by setting the option on the HTTP client itself `WithRandomTLSExtensionOrder()`.\ This is needed for Chrome 107+. #### Shared Library & Standalone Api When working with the Shared Library or Standalone API you need to set `"withRandomExtensionOrder":true` # Installation & Quick Usage ## Installation ```go go get -u github.com/bogdanfinn/tls-client // or specific version: // go get github.com/bogdanfinn/tls-client@v1.7.2 ``` Some users have trouble when using `go get -u`. If this is the case for you please cleanup your go.mod file and do a `go get` with a specific version like described above in the commented out code. I would recommend to check the Github tags for the latest version and install that one explicit. ## Quick Usage Example ```go package main import ( "fmt" "io" "log" http "github.com/bogdanfinn/fhttp" "github.com/bogdanfinn/tls-client/profiles" tls_client "github.com/bogdanfinn/tls-client" ) func main() { jar := tls_client.NewCookieJar() options := []tls_client.HttpClientOption{ tls_client.WithTimeoutSeconds(30), tls_client.WithClientProfile(profiles.Chrome_120), tls_client.WithNotFollowRedirects(), tls_client.WithCookieJar(jar), // create cookieJar instance and pass it as argument //tls_client.WithProxyUrl("http://user:pass@host:port"), //tls_client.WithInsecureSkipVerify(), } client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), options...) if err != nil { log.Println(err) return } req, err := http.NewRequest(http.MethodGet, "https://tls.peet.ws/api/all", nil) if err != nil { log.Println(err) return } req.Header = http.Header{ "accept": {"*/*"}, "accept-language": {"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, "user-agent": {"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36"}, http.HeaderOrderKey: { "accept", "accept-language", "user-agent", }, } resp, err := client.Do(req) if err != nil { log.Println(err) return } defer resp.Body.Close() log.Println(fmt.Sprintf("status code: %d", resp.StatusCode)) readBytes, err := io.ReadAll(resp.Body) if err != nil { log.Println(err) return } log.Println(string(readBytes)) } ``` For more configured clients check this documentation or use your own custom client. See provided examples how to use a complete custom TLS client. # Client Options When instantiating the TLS client you can define various options which are documented here: * WithClientProfile ``` WithClientProfile configures a TLS client to use the specified client profile. ``` * WithForceHttp1 ``` WithForceHttp1 configures a client to force HTTP/1.1 as the used protocol. ``` * WithDisableHttp3 ``` WithDisableHttp3 configures a client to disable HTTP 3 as the used protocol. Will most likely fall back to HTTP 2 ``` * WithInsecureSkipVerify ``` WithInsecureSkipVerify configures a client to skip SSL certificate verification. ``` * WithTransportOptions ``` WithTransportOptions configures a client to use the specified transport options. ``` The `TransportOptions` struct supports the following fields for client certificate authentication: ```go tls_client.WithTransportOptions(&tls_client.TransportOptions{ // Certificates is a list of client TLS certificates for mutual TLS (mTLS). // Load certificates with tls.LoadX509KeyPair or tls.X509KeyPair. Certificates: []tls.Certificate{clientCert}, }) ``` * WithProxyUrl ``` WithProxyUrl configures a HTTP client to use the specified proxy URL. proxyUrl should be formatted as: "http://user:pass@host:port" ``` * WithCharlesProxy ``` WithCharlesProxy configures the HTTP client to use a local running charles as proxy. host and port can be empty, then default 127.0.0.1 and port 8888 will be used ``` * WithCookieJar ``` WithCookieJar configures a HTTP client to use the specified cookie jar. ``` * WithServerNameOverwrite ``` WithServerNameOverwrite configures a TLS client to overwrite the server name being used for certificate verification and in the client hello. This option does only work properly if WithInsecureSkipVerify is set to true in addition ``` * WithTimeoutMilliseconds ``` WithTimeoutMilliseconds configures a hard deadline for the entire request lifecycle. This includes connection time, redirects, and reading the response body. WARNING: If the timer expires, the connection is forcibly closed, even if you are actively downloading data. - Use 0 to disable the deadline (unlimited) for large downloads or long-polling. - Default is 30000 milliseconds (30 seconds). ``` * WithTimeoutSeconds ``` WithTimeoutSeconds configures a hard deadline for the entire request lifecycle. This includes connection time, redirects, and reading the response body. WARNING: If the timer expires, the connection is forcibly closed, even if you are actively downloading data. - Use 0 to disable the deadline (unlimited) for large downloads or long-polling. - Default is 30 seconds. ``` * WithTimeout ``` WithTimeout configures an HTTP client to use the specified request timeout. timeout is the request timeout in seconds. Deprecated: use either WithTimeoutSeconds or WithTimeoutMilliseconds ``` * WithNotFollowRedirects ``` WithNotFollowRedirects configures an HTTP client to not follow HTTP redirects. ``` * WithCustomRedirectFunc ``` WithCustomRedirectFunc configures an HTTP client to use a custom redirect func. The redirect func have to look like that: func(req *http.Request, via []*http.Request) error Please only provide a custom redirect function if you know what you are doing. Check docs on net/http.Client CheckRedirect ``` * WithRandomTLSExtensionOrder ``` WithRandomTLSExtensionOrder configures a TLS client to randomize the order of TLS extensions being sent in the ClientHello. Placement of GREASE and padding is fixed and will not be affected by this. ``` * WithCertificatePinning ``` WithCertificatePinning enables SSL Pinning for the client and will throw an error if the SSL Pin is not matched. Please refer to https://github.com/tam7t/hpkp/#examples in order to see how to generate pins. The certificatePins are a map with the host as key. You can provide a BadPinHandlerFunc or nil as second argument. This function will be executed once a bad ssl pin is detected. ``` * WithDebug ``` WithDebug configures a client to log debugging information. ``` * WithLocalAddr ``` WithLocalAddr configures an HTTP client to use the specified local address. ``` * WithDisableIPV4 ``` WithDisableIPV4 configures a dialer to use tcp6 network argument ``` * WithDisableIPV6 ``` WithDisableIPV6 configures a dialer to use tcp4 network argument ``` * WithCatchPanics ``` WithCatchPanics configures a client to catch all go panics happening during a request and not print the stacktrace. ``` * WithDefaultHeaders ``` WithDefaultHeaders configures a client to use a set of default headers if none are specified on the request. ``` * WithConnectHeaders ``` WithConnectHeaders configures a client to use the specified headers for the CONNECT request ``` * WithProtocolRacing ``` WithProtocolRacing configures a client to race HTTP/3 (QUIC) and HTTP/2 (TCP) connections in parallel. Similar to Chrome's "Happy Eyeballs" approach, this starts both connection types simultaneously and uses whichever connects first. The client will remember which protocol worked for each host and use it directly on subsequent requests. This option is ignored if WithForceHttp1 or WithDisableHttp3 is set. ``` * WithBandwidthTracker ``` WithBandwidthTracker configures a client to track the bandwidth used by the client. You can retrieve the tracker via GetBandwidthTracker() and call GetTotalBandwidth(), GetWriteBytes(), GetReadBytes() or Reset() on it. ``` * WithDialer ``` WithDialer configures an HTTP client to use the specified dialer. This allows the use of a custom DNS resolver. ``` * WithProxyDialerFactory ``` WithProxyDialerFactory configures an HTTP client to use a custom proxyDialerFactory instead of the default newConnectDialer(). This allows to implement custom proxy dialer use cases. ``` * WithDialContext ``` WithDialContext sets a custom dialer for TCP connections, allowing advanced networking (Zero-DNS, socket tagging, DPI bypass). WARNING: This overrides built-in proxy settings. If you need a proxy, you must handle the CONNECT handshake manually. ``` Example: ```go client, err := tls_client.NewHttpClient( tls_client.NewNoopLogger(), tls_client.WithClientProfile(profiles.Chrome_124), tls_client.WithDialContext(func(ctx context.Context, network, addr string) (net.Conn, error) { // Custom dialing logic here return net.Dial(network, addr) }), ) ``` * WithPreHook ``` WithPreHook adds a pre-request hook that is called before each request is sent. Multiple hooks can be added and they will be executed in the order they were added. If any hook returns an error, the request is aborted and subsequent hooks are not called. To continue hook execution despite an error, wrap your error with ErrContinueHooks. ``` The hook function signature is: ```go type PreRequestHookFunc func(req *http.Request) error ``` Example: ```go client, err := tls_client.NewHttpClient( tls_client.NewNoopLogger(), tls_client.WithClientProfile(profiles.Chrome_124), tls_client.WithPreHook(func(req *http.Request) error { req.Header.Set("X-Custom-Header", "my-value") return nil }), ) ``` You can also add hooks at runtime using `client.AddPreRequestHook(hook)`. * WithPostHook ``` WithPostHook adds a post-response hook that is called after each request completes. Multiple hooks can be added and they will be executed in the order they were added. Hooks receive a PostResponseContext containing the request, response, and any error that occurred. If a hook returns an error, subsequent hooks are not called (unless wrapped with ErrContinueHooks). ``` The hook function signature and context struct are: ```go type PostResponseContext struct { Request *http.Request Response *http.Response Error error // Non-nil if request failed } type PostResponseHookFunc func(ctx *PostResponseContext) error ``` Example: ```go client, err := tls_client.NewHttpClient( tls_client.NewNoopLogger(), tls_client.WithClientProfile(profiles.Chrome_124), tls_client.WithPostHook(func(ctx *tls_client.PostResponseContext) error { if ctx.Error != nil { log.Printf("Request failed: %v", ctx.Error) } else { log.Printf("Response status: %d", ctx.Response.StatusCode) } return nil }), ) ``` You can also add hooks at runtime using `client.AddPostResponseHook(hook)`. * ErrContinueHooks ``` ErrContinueHooks can be returned (or wrapped) by a hook function to signal that the error should be logged but hook execution should continue to the next hook. By default any error returned from a hook aborts subsequent hooks (and for pre-hooks, the request). ``` Example: ```go tls_client.WithPreHook(func(req *http.Request) error { // This error will be logged but subsequent hooks and the request will continue return fmt.Errorf("something went wrong: %w", tls_client.ErrContinueHooks) }) ``` #### Shared Library & Standalone API When using the shared library or standalone api you can find almost every client option as a separate field to turn on / turn off or configure in the request payload. Please take a look at the Documentation about the [Payload](/open-source-oasis/shared-library/payload). # Cookiejar Since version 1.0.0 the TLS-Client uses the custom cookiejar implemented which can be found here: \ Before it was more or less golangs default cookiejar: The custom cookiejar is designed for some specific (sneaker)botting related use cases when it comes to cookie overwriting or sharing cookies across different TLD. Often these use cases were about overwriting exisiting cookies in the jar with newer values or something like that. When using the TLS-Client in go you can easilly switch between both jars just by instantiating the specific cookiejar you need: #### Default Go Cookiejar ``` jar, _ := cookiejar.New(nil) options := []tls_client.HttpClientOption{ tls_client.WithCookieJar(jar), } client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), options...) ``` #### Custom Cookiejar ``` jar := tls_client.NewCookieJar() options := []tls_client.HttpClientOption{ tls_client.WithCookieJar(jar), } client, err := tls_client.NewHttpClient(logger, options...) ``` #### Shared Library & Standalone API If you are using the shared library or standalone api you can either completely deactiavte cookie jar handling by setting `"withoutCookieJar"` to `true` or you can switch to the default Go cookiejar by setting `"withDefaultCookieJar"` to `true`. By default the Custom Cookiejar will always be used. # Defaults If you call `ProvideDefaultClient()` you will receive a TLS-Client instance with the following settings * 30 seconds timeout * Chrome 146 as profile * Random TLS Extension order enabled * Follow Redirects disabled * Idle Connections will be closed after 90 seconds # Request Headers ### Header Sorting You can sort your request headers per request by just providing a special header key (`"Header-Order:"`) with a list of header keys as sort order. Please keep in mind that it does not matter if your header keys are uppercase or all lowercase. For the header order **you have to define the header keys in the order list all lowercase.** ```go req.Header = http.Header{ "header4": {`value4`}, "header2": {"value2"}, "header1": {"value1"}, "header3": {"value3"}, "Header-Order:": { "header1", "header2", "header3", "header4", }, } ``` #### Shared Library & Standalone API When you are using the shared library or the standalone api this works exactly the same. You can provide the header order with the same Header Order key. ### Header Key Capitalization Sometimes people are confused when to write header keys all lowercase and when to write them with first uppercase letter. For HTTP2 requests header keys are usually all lowercase. For HTTP1 Requests header keys are usually first uppercase letter. Just as a rule of thumb: `"accept-encoding"` => HTTP2 `"Accept-Encoding"` => HTTP1 # Pseudo Header Order Pseudo headers are the following headers: `":method"`, `":authority"`, `":scheme"`, `":path"` and the order of them varies per client profile. Therefore you can define the order of the pseudo headers on the (custom-)client Profile. ```go pseudoHeaderOrder := []string{ ":method", ":authority", ":scheme", ":path", } customClientProfile := tls_client.NewClientProfile(tls.ClientHelloID{ Client: "MyCustomProfile", Version: "1", Seed: nil, SpecFactory: specFunc, }, settings, settingsOrder, pseudoHeaderOrder, connectionFlow, nil, nil) ``` #### Shared Library & Standalone API You will see on the CustomClient object [payload](/open-source-oasis/shared-library/payload) that you can define the order as a list of strings. # Proxies Proxies can be used by either setting them as client option `WithProxyUrl` when creating the TLS client or calling `SetProxy` on an exisiting client instance. A proxy should be formatted like this: `http://username:password@host:port` or `socks5://username:password@host:port` If you are using rotating Proxies (proxies with a static URL) then make sure to always call SetProxy with the same proxy again to let the client reconnect to the Proxy Server and receive a new IP-Address. #### Shared Library & Standalone API When you are using the shared library or the standalone api you can supply a proxy in the `"proxyUrl"` field and set `"isRotatingProxy"` to `true` or `false`. # Certificate Pinning You can enable certificate pinning for a TLS-Client instance. You need to provide a map of pins by host when you create the client. See the example code below ``` pins := map[string][]string{ "bstn.com": { "NQvy9sFS99nBqk/nZCUF44hFhshrkvxqYtfrZq3i+Ww=", "4a6cPehI7OG6cuDZka5NDZ7FR8a60d3auda+sKfg4Ng=", "x4QzPSC810K5/cMjb05Qm4k3Bw5zBn4lTdO/nEW/Td4=", }, } options := []tls_client.HttpClientOption{ tls_client.WithCertificatePinning(pins, tls_client.DefaultBadPinHandler), } client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), options...) ``` This example code shows how to enabled certificate pinning for bstn.com. Next to it you can supply a BadPinHandler function as second argument for the WithCertificatePinning option. The default BadPinHandler looks like this. This function will be executed when a bad pin is detected: ``` var DefaultBadPinHandler = func(req *http.Request) { fmt.Println("this is the default bad pin handler") } ``` #### Shared Library & Standalone API When using the shared library you can supply certificate pinning settings via the `"certificatePinningHosts"` field. This should look similar to this: ``` "certificatePinningHosts": { "bstn.com": [ "NQvy9sFS99nBqk/nZCUF44hFhshrkvxqYtfrZq3i+Ww=", "4a6cPehI7OG6cuDZka5NDZ7FR8a60d3auda+sKfg4Ng=", "x4QzPSC810K5/cMjb05Qm4k3Bw5zBn4lTdO/nEW/Td4=", ], }, ``` #### Wildcards You can define wildcards for subdomains for example like this ``` pins := map[string][]string{ "*.bstn.com": { "NQvy9sFS99nBqk/nZCUF44hFhshrkvxqYtfrZq3i+Ww=", "4a6cPehI7OG6cuDZka5NDZ7FR8a60d3auda+sKfg4Ng=", "x4QzPSC810K5/cMjb05Qm4k3Bw5zBn4lTdO/nEW/Td4=", }, } options := []tls_client.HttpClientOption{ tls_client.WithCertificatePinning(pins, tls_client.DefaultBadPinHandler), } client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), options...) ``` #### How to generate pins? You can easilly generate pins with this helpful tool: \ Just install the tool and run the following command against the site you want to generate pins for: ``` hpkp-pins -server=bstn.com:443 ``` # Response Body Encoding / Decoding If you are specifying `accept-encoding` header yourself and you are on a `http1` connection than you have to take care of the response body decompression yourself. It is not done automatically. Only if you are **not** adding `accept-encoding` header then the library adds it for you if not explicit disabled and also handles the decompression automatically. On `http2` the automatic decompression should always be in place according to the `content-type` Header on the Response. `DecompressBody` is an exported function you can use. See the following example on how do decompress the response body manually ```go req, err := http.NewRequest(http.MethodGet, "https://tls.browserleaks.com/json", nil) if err != nil { log.Println(err) goreturn } req.Header = http.Header{ "accept": {"*/*"}, "accept-encoding": {"gzip"}, "accept-language": {"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, "user-agent": {"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"}, http.HeaderOrderKey: { "accept", "accept-encoding", "accept-language", "user-agent", }, } resp, err := client.Do(req) if err != nil { log.Println(err) return } defer resp.Body.Close() decomBody := http.DecompressBody(resp) all, err := ioutil.ReadAll(decomBody) if err != nil { log.Println(err) return } log.Println(string(all)) ``` It is possible to disable the automatic decompression on http2 by adding the following option to the http client ```go tls_client.WithTransportOptions(&tls_client.TransportOptions{ DisableCompression: true, }), ``` #### Automatic Charset Detection When using the shared library or standalone API, the response body charset is automatically detected and decoded based on the `Content-Type` response header (e.g. `charset=euc-kr`, `charset=shift_jis`, etc.). This means you no longer need to handle specific encodings manually — the correct character encoding is applied automatically for all text responses. This behavior is active whenever `isByteResponse` is `false` (the default). If you need the raw bytes, set `isByteResponse: true`. #### Shared Library & Standalone API When you are using the shared library or standalone api application the response body will always be decompressed and the charset will be automatically detected and decoded. # Custom Client Profile **You should know at least a bit about TLS Fingerprints in order to know what you are doing here and what is minimum required.** You can build complete custom clients which consist of different things. When we look at the factory method we see that you need to provide the following information: * A clientHelloId which holds the `ClientHelloSpecFactory`. More about that later. * A key value map of http2 settings * A list of http2 settings for the order * A list of [pseudo headers](/open-source-oasis/readme/pseudo-header-order) for the order * A value for the connection flow * A slice of http2 priority settings * A optional header priority setting ```go func NewClientProfile(clientHelloId tls.ClientHelloID, settings map[http2.SettingID]uint32, settingsOrder []http2.SettingID, pseudoHeaderOrder []string, connectionFlow uint32, priorities []http2.Priority, headerPriority *http2.PriorityParam) ClientProfile { return ClientProfile{ clientHelloId: clientHelloId, settings: settings, settingsOrder: settingsOrder, pseudoHeaderOrder: pseudoHeaderOrder, connectionFlow: connectionFlow, priorities: priorities, headerPriority: headerPriority, } } ``` You can create a `ClientHelloSpecFactory` out of a ja3 string by calling GetSpecFactoryFromJa3String. You need to provide in addition the following information, as these can't be derived from the ja3 string: * A list of supported signature algorithms * A list of supported delegated credentials algorithms * A list of supported TLS versions * A list of supported key share curves * A list of certificate compression algorithms. * A value for the RecordSizeLimit Extension ```go ja3 := "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-10-11-13-16-23-43-51-65281-45-21,29-23-24,0" ssa := []string{"ECDSAWithP256AndSHA256", "PSSWithSHA256", "PKCS1WithSHA256", "ECDSAWithP384AndSHA384", "PSSWithSHA384", "PKCS1WithSHA384", "PSSWithSHA512", "PKCS1WithSHA512"} dca := []string{"ECDSAWithP256AndSHA256", "PSSWithSHA256", "PKCS1WithSHA256", "ECDSAWithP384AndSHA384", "PSSWithSHA384", "PKCS1WithSHA384", "PSSWithSHA512", "PKCS1WithSHA512"} sv := []string{"GREASE", "1.3", "1.2"} sc := []string{"GREASE", "X25519"} alpnProtocols := []string{"h2", "http/1.1"} alpsProtocols := []string{"h2"} ccs := []tls_client.CandidateCipherSuites{ { KdfId: "HKDF_SHA256", AeadId: "AEAD_AES_128_GCM", }, { KdfId: "HKDF_SHA256", AeadId: "AEAD_CHACHA20_POLY1305", }, } cp := []uint16{128, 160, 192, 224} certCompressionAlgos := []string{"zlib"} recordSizeLimit := 0 specFunc, err := tls_client.GetSpecFactoryFromJa3String(input, ssa, dca, sv, sc, alpnProtocols, alpsProtocols, ccs, cp, certCompressionAlgos, recordSizeLimit) ``` At the end you can just build the complete custom profile by providing all the above mentioned information. ```go ja3 := "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-10-11-13-16-23-43-51-65281-45-21,29-23-24,0" ssa := []string{"ECDSAWithP256AndSHA256", "PSSWithSHA256", "PKCS1WithSHA256", "ECDSAWithP384AndSHA384", "PSSWithSHA384", "PKCS1WithSHA384", "PSSWithSHA512", "PKCS1WithSHA512"} dca := []string{"ECDSAWithP256AndSHA256", "PSSWithSHA256", "PKCS1WithSHA256", "ECDSAWithP384AndSHA384", "PSSWithSHA384", "PKCS1WithSHA384", "PSSWithSHA512", "PKCS1WithSHA512"} sv := []string{"GREASE", "1.3", "1.2"} sc := []string{"GREASE", "X25519"} alpnProtocols := []string{"h2", "http/1.1"} alpsProtocols := []string{"h2"} ccs := []tls_client.CandidateCipherSuites{ { KdfId: "HKDF_SHA256", AeadId: "AEAD_AES_128_GCM", }, { KdfId: "HKDF_SHA256", AeadId: "AEAD_CHACHA20_POLY1305", }, } cp := []uint16{128, 160, 192, 224} certCompressionAlgos := []string{"zlib"} recordSizeLimit := 0 specFunc, err := tls_client.GetSpecFactoryFromJa3String(input, ssa, dca, sv, sc, alpnProtocols, alpsProtocols, ccs, cp, certCompressionAlgos, recordSizeLimit) if err != nil { log.Println(err.Error()) } settings := map[http2.SettingID]uint32{ http2.SettingHeaderTableSize: 65536, http2.SettingMaxConcurrentStreams: 1000, http2.SettingInitialWindowSize: 6291456, http2.SettingMaxHeaderListSize: 262144, } settingsOrder := []http2.SettingID{ http2.SettingHeaderTableSize, http2.SettingMaxConcurrentStreams, http2.SettingInitialWindowSize, http2.SettingMaxHeaderListSize, } pseudoHeaderOrder := []string{ ":method", ":authority", ":scheme", ":path", } connectionFlow := uint32(15663105) customClientProfile := tls_client.NewClientProfile(tls.ClientHelloID{ Client: "MyCustomProfile", Version: "1", Seed: nil, SpecFactory: specFunc, }, settings, settingsOrder, pseudoHeaderOrder, connectionFlow, nil, nil) ``` #### Shared Library & Standalone API It is also possible to define custom client profiles when working with the shared library or the standalone api. For that you are not calling functions but provide all the information on the request payload that the TLS client is able to build the client profile out of it. Instead of the `"tlsClientIdentifier"` you can provide on the request payload the `"customTlsClient"`. ```json "customTlsClient": { "ja3String": "771,2570-4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,2570-0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-2570-21,2570-29-23-24,0", "h2Settings": { "HEADER_TABLE_SIZE": 65536, "MAX_CONCURRENT_STREAMS": 1000, "INITIAL_WINDOW_SIZE": 6291456, "MAX_HEADER_LIST_SIZE": 262144 }, "h2SettingsOrder": [ "HEADER_TABLE_SIZE", "MAX_CONCURRENT_STREAMS", "INITIAL_WINDOW_SIZE", "MAX_HEADER_LIST_SIZE" ], "supportedSignatureAlgorithms": [ "ECDSAWithP256AndSHA256", "PSSWithSHA256", "PKCS1WithSHA256", "ECDSAWithP384AndSHA384", "PSSWithSHA384", "PKCS1WithSHA384", "PSSWithSHA512", "PKCS1WithSHA512", ], "supportedVersions": ["GREASE", "1.3", "1.2"], "keyShareCurves": ["GREASE", "X25519"], "ECHCandidatePayloads": [], "certCompressionAlgos": ["brotli"], "recordSizeLimit": 0, "pseudoHeaderOrder": [ ":method", ":authority", ":scheme", ":path" ], "connectionFlow": 15663105, "priorityFrames": [{ "streamID": 1, "priorityParam": { "streamDep": 1, "exclusive": true, "weight": 1 } }], "headerPriority": { "streamDep": 1, "exclusive": true, "weight": 1 }, }, ``` # WebSocket The TLS-Client supports WebSocket connections with the same TLS fingerprinting as regular HTTP requests. This ensures consistent fingerprinting across both HTTP and WebSocket connections. **Important:** WebSocket connections require HTTP/1.1. You **must** use `WithForceHttp1()` when creating the HTTP client. ### Usage (Go) ```go // Create HTTP client with ForceHttp1 (required for WebSocket!) client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), tls_client.WithClientProfile(profiles.Chrome_133), tls_client.WithForceHttp1(), ) // Create WebSocket with optional header ordering headers := http.Header{ "User-Agent": {"MyBot/1.0"}, http.HeaderOrderKey: {"host", "upgrade", "connection", "user-agent"}, } ws, err := tls_client.NewWebsocket(nil, tls_client.WithUrl("wss://example.com/ws"), tls_client.WithTlsClient(client), tls_client.WithHeaders(headers), ) conn, err := ws.Connect(context.Background()) defer conn.Close() ``` ### WebSocket Options * WithUrl ``` WithUrl sets the WebSocket URL to connect to. Required. ``` * WithTlsClient ``` WithTlsClient sets the tls-client HttpClient to use for the WebSocket connection. The underlying dialer from this client will be used to establish the connection, preserving TLS fingerprinting and other client configurations. Required. ``` * WithHeaders ``` WithHeaders sets the HTTP headers to send during the WebSocket handshake. You can use the http.HeaderOrderKey to control header ordering. ``` * WithReadBufferSize ``` WithReadBufferSize sets the read buffer size for the WebSocket connection. ``` * WithWriteBufferSize ``` WithWriteBufferSize sets the write buffer size for the WebSocket connection. ``` * WithHandshakeTimeoutMilliseconds ``` WithHandshakeTimeoutMilliseconds sets the timeout for the WebSocket handshake in milliseconds. ``` * WithCookiejar ``` WithCookiejar sets the cookie jar to use for the WebSocket connection. ``` # Protocol Racing Protocol Racing allows the TLS-Client to race HTTP/3 (QUIC) and HTTP/2 (TCP) connections in parallel, similar to Chrome's "Happy Eyeballs" approach. Both connection types are started simultaneously and whichever connects first is used. The client remembers which protocol worked for each host and uses it directly on subsequent requests. If the cached protocol fails, the client falls back to racing again. ### How it works 1. On the first request to a host, both HTTP/3 and HTTP/2 connections are started in parallel. 2. HTTP/2 is delayed by 300ms (same as Chrome) to give HTTP/3 a slight head start. 3. Whichever protocol succeeds first is used for the request. 4. The winning protocol is cached for the host and used directly on future requests. 5. If the cached protocol fails, the cache is cleared and racing restarts. ### Usage (Go) ```go client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), tls_client.WithClientProfile(profiles.Chrome_133), tls_client.WithProtocolRacing(), ) ``` ### Configuration Constraints Protocol Racing cannot be used in combination with certain other options. The following configurations are invalid and will result in an error: * `WithProtocolRacing()` + `WithDisableHttp3()` — Racing requires HTTP/3 to be enabled. * `WithProtocolRacing()` + `WithForceHttp1()` — Racing requires HTTP/2 and HTTP/3. * `WithDisableIPV4()` + `WithDisableIPV6()` — Cannot disable both IP versions. * `WithCertificatePinning()` + `WithInsecureSkipVerify()` — Certificate pinning is incompatible with skipping verification. #### Shared Library & Standalone API When using the shared library or standalone API, set `"withProtocolRacing": true` in the request payload to enable protocol racing. The same configuration constraints apply. # Examples Please take a look at the following Go Code in the repository to see examples implemented: `./example/main.go` We provide examples for the following use cases: * GET Request * POST Request * Proxy Rotation * Custom Client Profile * Switching the Redirect Following Behavior * SSL Pinning * Downloading Image / Files # Shared Library # Node Version Some users experienced issues when using the shared library with NodeJS14 or lower. It is recommended to use at least NodeJS16. # Downloads Here you can find the latest prebuilt shared library. Older versions before 1.3.8 can be found in the Github history of the repository: # Build from source When you want to build the shared libraries from source you can just run the following script: `cffi_dist/build.sh SOME_BUILD_IDENTIFIER` and it should build the binaries for you. # Exposed Methods The shared library exposes the following endpoints / methods you can call. The payload is always a JSON string expect for the `freeMemory` method. You will find the methods also in use in the examples. #### request(payload: string) => string This is the basic method you will use to do requests. For full detailed documentation about the payload please take a look at [Payload](/open-source-oasis/shared-library/payload). As you see here on other methods you have to turn the JSON Object into a JSON string before supplying it as the payload parameter. #### getCookiesFromSession(payload: string) => string Get all cookies which are in the provided session for a given URL. The payload should look like this: ```json "{\"sessionId\": \"someExistingSessionId\", \"url\": \"urlToGetCookiesFor\"}" ``` #### addCookiesToSession(payload: string) => string Add manual cookies to an existing session. The payload should look like this: ```json "{\"sessionId\": \"someExistingSessionId\", \"url\": \"urlToAddCookiesFor\", \"cookies\": []}" ``` #### freeMemory(responseId: string) => void Supply the `"id"` of a previous response as string in order to free the allocated memory of the previous response. #### destroyAll() => string Destroy all existing sessions in order to release allocated memory. #### destroySession(payload: string) => string Destroy a specific session in order to release allocated memory. The payload should look like this: ```json "{\"sessionId\": \"someExistingSessionId\"}" ``` # JavaScript # Examples Please take a look at the following NodeJS Code in the repository to see examples implemented: `cffi_dist/example_node/` We provide examples for the following use cases: * GET Request * POST Request * Image Download * Image Upload * Async Request * Custom Client * Use Cookies * Use ffi-rs instead of ffi-napi # Python # Examples Please take a look at the following Python Code in the repository to see examples implemented: `cffi_dist/example_python/` We provide examples for the following use cases: * GET Request * POST Request * Image Download * Image Upload * Async Request * Custom Client * Use Cookies # TypeScript # Examples Please take a look at the following Typescript Code in the repository to see examples implemented: `cffi_dist/example_typescript/` We provide examples for the following use cases: * GET Request * Async Request In general you can also take a look at the [JavaScript Examples](/open-source-oasis/shared-library/javascript/examples). The Typescript examples are more about how to define the types correctly and use them. # C\# # Examples Please take a look at the following C# Code in the repository to see examples implemented: `cffi_dist/example_csharp/` We provide examples for the following use cases: * Loading and definition of the shared Library # Defaults The client from the shared library uses the same default settings as the regular TLS client which are defined [here](/open-source-oasis/readme/defaults) # Payload This page shows once the full possible request payload against the shared library, with all default values. #### Basic Request Input ``` { "catchPanics": false, "certificatePinningHosts": null, "customTlsClient": null, "transportOptions": null, "followRedirects": false, "forceHttp1": false, "disableHttp3": false, "withProtocolRacing": false, "headerOrder": null, "headers": null, "insecureSkipVerify": false, "isByteRequest": false, "isByteResponse": false, "isRotatingProxy": false, "proxyUrl": null, "requestBody": null, "requestCookies": null, "requestHostOverride": null, "defaultHeaders": null, "connectHeaders": null, "requestMethod": "", "requestUrl": "", "disableIPV6": false, "disableIPV4": false, "localAddress": null, "sessionId": null, "serverNameOverwrite": "", "streamOutputBlockSize": null, "streamOutputEOFSymbol": null, "streamOutputPath": null, "timeoutMilliseconds": 0, "timeoutSeconds": 0, "tlsClientIdentifier": "", "withDebug": false, "withCustomCookieJar": false, "withoutCookieJar": false, "withRandomTLSExtensionOrder": false } ``` | Field | Type | Description | | ------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | headers | Map\ | Headers to attach on the current request. defaultHeaders will be used when empty. | | defaultHeaders | Map\> | Default Headers to be used when no request headers are specified. The default header order can be specified with the Key "Header-Order:" | | connectHeaders | Map\> | Headers to be used during the CONNECT request. | | serverNameOverwrite | string | Lookup [Client Option ](/open-source-oasis/readme/client-options)"WithServerNameOverwrite" | * `sessionId` is optional. When not provided the API does not create a Session. On every forwarded request with a given sessionId you will receive the sessionId in the response to be able to reuse sessions (cookies). * Be aware that `insecureSkipVerify` and the `timeoutSeconds` can not be changed during a session. * `followRedirects` and `proxyUrl` can be changed within a session. * If you do not want to set `requestBody` or `proxyUrl` use `null` instead of empty string * When you set `isByteResponse` to `true` the response body will be a base64 encoded string. Useful when you want to download images for example. * When you set `isByteRequest` to `true` the request body needs to be a base64 encoded string. Useful when you want to upload images for example. * When you set `withProtocolRacing` to `true` the client will race HTTP/3 (QUIC) and HTTP/2 (TCP) connections in parallel, similar to Chrome's "Happy Eyeballs" approach. Cannot be used together with `forceHttp1` or `disableHttp3`. * When you set `withCustomCookieJar` to `true` a custom TLS-Client cookie jar will be used which is more suited for certain use cases. Otherwise the default Go cookie jar is used. #### Custom TLS-Client ``` { "certCompressionAlgos": [], "connectionFlow": 0, "h2Settings": null, "h2SettingsOrder": null, "h3Settings": null, "h3SettingsOrder": null, "h3PseudoHeaderOrder": null, "h3PriorityParam": 0, "h3SendGreaseFrames": false, "headerPriority": null, "ja3String": "", "keyShareCurves": null, "priorityFrames": null, "alpnProtocols": null, "alpsProtocols": null, "ECHCandidatePayloads": null, "ECHCandidateCipherSuites": null, "pseudoHeaderOrder": null, "supportedDelegatedCredentialsAlgorithms": null, "supportedSignatureAlgorithms": null, "supportedVersions": null, "recordSizeLimit": 0, "streamId": 0, "allowHttp": false } ```
FieldTypeDescription
certCompressionAlgosArray<string>See possible values at the end of this page
connectionFlowinteger
h2SettingsMap<string, int>See possible values for the Map keys at the end of this page.
h2SettingsOrderArray<string>Array of string keys which are used in the h2Settings property but ordered.
h3SettingsMap<string, int>HTTP/3 settings. See possible values for the Map keys at the end of this page.
h3SettingsOrderArray<string>Array of string keys which are used in the h3Settings property but ordered.
h3PseudoHeaderOrderArray<string>Pseudo header order for HTTP/3 requests.
h3PriorityParamintegerHTTP/3 priority parameter.
h3SendGreaseFramesbooleanWhether to send GREASE frames in HTTP/3.
headerPriorityPriorityParamSee type definition below in next section
ja3Stringstring
keyShareCurvesArray<string>See possible values at the end of this page
priorityFramesArray<PriorityFrames>See type definition below in next section
alpnProtocolsArray<string>List of supported protocols for the ALPN Extension
alpsProtocolsArray<string>List of supported protocols for the ALPS Extension
ECHCandidatePayloadsArray<uint16>List of ECH Candidate Payloads
ECHCandidateCipherSuitesArray<CandidateCipherSuite>See type definition below in next section
pseudoHeaderOrderArray<string>See possible values at the end of this page
supportedDelegatedCredentialsAlgorithmsArray<string>See possible values at the end of this page
supportedSignatureAlgorithmsArray<string>See possible values at the end of this page
supportedVersionsArray<string>See possible values at the end of this page
recordSizeLimitintegerTLS record size limit extension value
streamIdintegerInitial HTTP/2 stream ID
allowHttpbooleanAllow plaintext HTTP connections
#### TransportOptions ``` { "disableKeepAlives": false, "disableCompression": false, "maxIdleConns": 0, "maxIdleConnsPerHost": 0, "maxConnsPerHost": 0, "maxResponseHeaderBytes": 0, "writeBufferSize": 0, "readBufferSize": 0, "idleConnTimeout": 0, } ``` | Field | Type | Description | | ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- | | disableKeepAlives | boolean | | | disableCompression | boolean | disables (automatic) decompression behavor as documented [here](/open-source-oasis/readme/response-body-encoding-decoding) | | maxIdleConns | integer | | | maxIdleConnsPerHost | integer | | | maxConnsPerHost | integer | | | maxResponseHeaderBytes | integer | If zero, a default is used | | writeBufferSize | integer | If zero, a default (currently 4KB) is used | | readBufferSize | integer | If zero, a default (currently 4KB) is used | | idleConnTimeout | integer | Duration in Nano Seconds | #### CandidateCipherSuite 
{
    "kdfId": "",
    "aeadId": "",
}
| Field | Type | Description | | ------ | ------ | ------------------------------------------- | | kdfId | string | See possible values at the end of this page | | aeadId | string | See possible values at the end of this page | #### PriorityParam 
{
    "streamDep": 0,
    "exclusive": false,
    "weight": 0
}
| Field | Type | Description | | --------- | ------- | ----------- | | streamDep | integer | | | exclusive | boolean | | | weight | integer | | #### PriorityFrames ``` { "streamID": 0, "priorityParam": null } ``` | Field | Type | Description | | ------------- | ------------- | ----------------------------- | | streamId | integer | | | priorityParam | PriorityParam | See type in the section above | | | | | #### Cookie Input ``` { "domain": "", "expires": 0, "maxAge": 0, "name": "", "path": "", "value": "", "secure": false, "httpOnly": false } ```
FieldTypeDescription
domainstring
expiresintegerUnix Timestamp
maxAgeintegerNumber of seconds the cookie is valid.
namestring
pathstring
valuestring
securebooleanWhether the cookie should only be sent over HTTPS.
httpOnlybooleanWhether the cookie is inaccessible to client-side scripts.
If both `Expires` and `Max-Age` are set, `Max-Age` has precedence. ### Values Here you can find the allowed possible string values to supply for fields like `supportedDelegatedCredentialsAlgorithms`, `supportedSignatureAlgorithms`, `supportedVersions` and much more. #### H2Settings ``` "HEADER_TABLE_SIZE", "ENABLE_PUSH", "MAX_CONCURRENT_STREAMS", "INITIAL_WINDOW_SIZE", "MAX_FRAME_SIZE", "MAX_HEADER_LIST_SIZE", "UNKNOWN_SETTING_7", "UNKNOWN_SETTING_8", "UNKNOWN_SETTING_9", ``` #### H3Settings ``` "QPACK_MAX_TABLE_CAPACITY", "MAX_FIELD_SECTION_SIZE", "QPACK_BLOCKED_STREAMS", "H3_DATAGRAM", ``` #### Supported Versions ``` "GREASE", "1.3", "1.2", "1.1", "1.0", ``` #### Supported Signature Algorithms ``` "PKCS1WithSHA256", "PKCS1WithSHA384", "PKCS1WithSHA512", "PSSWithSHA256", "PSSWithSHA384", "PSSWithSHA512", "ECDSAWithP256AndSHA256", "ECDSAWithP384AndSHA384", "ECDSAWithP521AndSHA512", "PKCS1WithSHA1", "ECDSAWithSHA1", "Ed25519", "SHA224_RSA", "SHA224_ECDSA", ``` #### certCompressionAlgorithm ``` "zlib", "brotli", "zstd", ``` #### Supported delegated credentials ``` "PKCS1WithSHA256", "PKCS1WithSHA384", "PKCS1WithSHA512", "PSSWithSHA256", "PSSWithSHA384", "PSSWithSHA512", "ECDSAWithP256AndSHA256", "ECDSAWithP384AndSHA384", "ECDSAWithP521AndSHA512", "PKCS1WithSHA1", "ECDSAWithSHA1", "Ed25519", "SHA224_RSA", "SHA224_ECDSA" ``` #### KeyShareCurves ``` "GREASE", "P256", "P384", "P521", "X25519", "P256Kyber768", "X25519Kyber512D", "X25519Kyber768", "X25519Kyber768Old", "X25519MLKEM768", ``` #### kdfIds ``` "HKDF_SHA256", "HKDF_SHA384", "HKDF_SHA512", ``` #### aeadIds ``` "AEAD_AES_128_GCM", "AEAD_AES_256_GCM", "AEAD_CHACHA20_POLY1305", ``` # Response The Response of a call against the shared library looks like this ``` { "id": "some response identifier", "sessionId": "some reusable sessionId if provided on the request", "status": 200, "target": "the target url", "body": "The Response as string here or the error message", "headers": {}, "cookies": {}, "usedProtocol": "h2" } ``` In case of an unexpected error the `"status"` will be 0 The `"id"` is necessary when you want to free the memory used to generate this response. Take a look at [Memory Issues](/open-source-oasis/shared-library/memory-issues). The `"sessionId"` can be reused to keep the cookies from the previous requests in this session. The `"status"` indicates the status code of the remote response. The `"body"` is the response body from the remote response. The `"headers"` are the headers returned from the remote response. The `"cookies"` are the cookies returned from the remote response. The `"usedProtocol"` indicates which protocol was used for the request (e.g. `"h2"`, `"HTTP/2.0"`, `"HTTP/1.1"`). # Memory Issues #### Go TLS Client & Standalone API For the pure go implementation there is no real possibility to run into memory leak issues, except of the fact that you implemented your request handling badly and the references to your objects are never really freed that the go garbage collection can not clean up the used memory. If you are running into memory leak issues with the Go client or the standalone api please double check your request handling in order so see if you are doing things not as recommended (for example not closing or deferred closing the response body inside an infinite loop) #### Shared Library If you are using the shared library there is potential to run into memory leak issues. Luckily you can avoid that by following a couple simple steps. By design there is memory allocated for every response coming from the go implementation (shared library) to the invoking application (python, node, etc.). The caller has to free the memory when he is done with handling the response. otherwise the memory will never be freed and you run into memory issues. On every response from the shared library you will see an `"id"` property. You can pass the value from the `"id"` field as parameter to the `freeMemory()` call of the shared library. ```javascript const response = tlsClientLibrary.request(JSON.stringify(requestPayload)); const responseObject = JSON.parse(response) tlsClientLibrary.freeMemory(responseObject.id) // free memory ``` ```python response = request(dumps(request_payload).encode('utf-8')) response_bytes = ctypes.string_at(response) response_string = response_bytes.decode('utf-8') response_object = loads(response_string) freeMemory(response_object['id'].encode('utf-8')) ``` # Standalone API Application The standalone API Application can be found here: This is an application which is using [gosoline](https://github.com/justtrackio/gosoline) and [TLS-Client](https://github.com/bogdanfinn/tls-client) to run a simple request forwarding service with the option to use specific TLS fingerprints which are implemented in [TLS-client](https://github.com/bogdanfinn/tls-client). # Download Here you can find the latest prebuilt binaries. Older versions before 1.3.8 can be found in the Github history of the repository: There is a binary for linux, macos and windows. Just modify the `config.dist.yml` file next to the binary to your needs and start the application. # Build from source When you want to build the application from source, make sure to also checkout this repository `https://github.com/Solem8s/gosoline` on the branch `tls-client-api` next to this project. \ Afterwards you can just run the following script: `cmd/tls-client-api/build.sh SOME_BUILD_IDENTIFIER` and it should build the binaries for you. # Configuration & Start * Configure things like api port and authentication keys in the `cmd/tls-client-api/config.dist.yml` file. * The default endpoint is `http://127.0.0.1:8080/api/forward` * You need to set a `x-api-key` header with an auth key from the config file. This is for protecting the API when you host it on some server. Requests without the correct keys in the header will be rejected. # Endpoints You need to do a POST Request against the running API with the following JSON Request Body. #### Forward Request `/api/forward` ``` curl --location --request POST '127.0.0.1:8080/api/forward' \ --header 'x-api-key: my-auth-key-1' \ --header 'Content-Type: application/json' \ --data-raw '{ "tlsClientIdentifier": "chrome_105", "requestUrl": "https://tls.peet.ws/api/all", "requestMethod": "GET" }' ``` #### Get Cookies from Session `/api/cookies` ``` curl --location --request POST '127.0.0.1:8080/api/cookies' \ --header 'x-api-key: my-auth-key-1' \ --header 'Content-Type: application/json' \ --data-raw '{ "sessionId":"my-custom-sessionid", "url":"http://some-url.com" }' ``` #### Free Single Session `/api/free-session` ``` curl --location --request POST '127.0.0.1:8080/api/free-session' \ --header 'x-api-key: my-auth-key-1' \ --header 'Content-Type: application/json' \ --data-raw '{ "sessionId":"my-custom-sessionid" }' ``` #### Free All Sessions `/api/free-all` ``` curl --location --request GET '127.0.0.1:8080/api/free-all' \ --header 'x-api-key: my-auth-key-1' ``` # Defaults For the standalone api the same defaults are configured as for the shared library you can find [here](/open-source-oasis/shared-library/defaults). # Attention * Applications powered with [gosoline](https://github.com/justtrackio/gosoline) automatically host a health check endpoint which is by default on port `8090` under the path `/health`. So in our case it would be `http://127.0.0.1:8090/health`. * Applications powered with [gosoline](https://github.com/justtrackio/gosoline) automatically host a metadata server for your application to provide insights into your application. The metadata server is hosted on port `8070` and has three endpoints. `/`, `/config`, `/memory` this should help you debugging your application. * **Do not make this endpoints public available when you host the Application on some server in the internet. You would make your config file public available.** # Payload The payload you can send against the standalone api is exactly the same as for the shared library which you can find [here](/open-source-oasis/shared-library/payload) # Response The response for the standalone api is the same as for the shared library which you can find [here](/open-source-oasis/shared-library/response) # How to get support # Frequently Asked Questions / Errors Here are some frequently asked questions about the cookiejar 1. How can i delete cookies from the session / jar? * Pass a new cookiejar instance to the clients SetCookiejar() method * For the shared library there is no option to delete all cookies of a session * You can delete a single cookie inside a session if you give the cookie a negative maxAge value. (Does only work for the tls\_client cookiejar not the standard cookiejar) 2. I can not do a successful POST Request with the shared library or standalone api * Be aware that when you do a POST Request and want to provide a forwarded request body in the `requestBody` field it has to be a string. That means if you want to send JSON you need to stringify this JSON to a string first and set the correct `content-type` header. 3. How can I use other request body content types besides json? * `requestBody` accepts strings and forwards them as the payload. combined with the `content-type` header the api makes the actual request body out of it. You can use for example `application/x-www-form-urlencoded` content type in the header and then just provide as request body a string similar to `key=value&key=value` 4. x509: certificate signed by unknown authority * As the message indicates you are trying to request a server which most likely has a self signed ssl certificate. There is a client option to skip the cerification. 5. x509: certificate has expired or is not yet valid * As the message indicates you are trying to request a server which most likely has an expired or invalid ssl certificate. There is a client option to skip the cerification. 6. stream error: stream ID 3; PROTOCOL\_ERROR * Sometimes this happens when people set a wrong `content-length` header on the request. Usually you never need to set the `content-length` header because the client will take care of that for you. # Community Support Please join the community support server here: and ask for support in the community. I will not give support to you in my Discord DMs. # Further Information This library uses the following api: to verify the hashes and fingerprints for http2 and ja3. Be aware that also peets api does not show every extension/cipher a TLS client is using. Do not rely just on ja3 strings. The API does not include the GREASE Extenson into the ja3 string. It’s omitted on purpose. Always have a look at the extension list and not only on the ja3 string. Show some love for peet here: If you appreciate my work feel free to [buy me a coffee](https://www.buymeacoffee.com/CaptainBarnius) or donate some money to charity. # Antibots & Captchas The TLS-Client is just a tool to mimic specific client profiles. It does not solve / pass antibot solutions which have other "challenges" like a javascript challenge or require solving a captcha. Luckily for this there are various services on the market to help you. #### HyperSolutions Hyper Solutions offers proven solutions for handling Akamai, Incapsula, DataDome, Kasada, and Cloudflare antibot systems. Our request-based APIs generate tokens and sensor data instantly. No browsers, no complexity, just fast and reliable bypass solutions trusted by leading companies across industries. {% embed url="" %} #### **ParallaxSystems** ParallaxSystems offers the most reliable solutions for handling various AntiBot Systems, trusted by leading companies across industries. With years of expertise, we ensure seamless integration and fast results, including direct cookie returns for ultimate efficiency. {% embed url="" %} #### Capsolver.com Capsolver‘s automatic captcha solver offers the most affordable and quick captcha-solving solution. You may rapidly combine it with your program using its simple integration option to achieve the best results in a matter of seconds. {% embed url="" %} #### Nstbrowser.io Nstbrowser is a powerful anti-detect browser designed for multi-accounting professionals to manage multiple accounts securely and efficiently. {% embed url="" %} #### Other services {% embed url="" %} # Proxies For efficient web scparing it is mandatory to utilize good proxies. Here are a list of a few providers you could use. #### Thordata Thordata is a leading global residential proxy provider with a massive pool of 60 million dynamic IPs. We empower large-scale web data extraction with fast, stable, and highly scalable access. With a guaranteed 99.9% uptime, Thordata offers a versatile range of Mobile, Static ISP, and Datacenter proxies. {% embed url="" %} # Community Projects Here you can find a list of projects which are using the TLS-Client ### Go * ### Python * * * * ### JavaScript / Node * * * ### Others * * # Contributing Contributions to this documentation are welcome! If you find something that is outdated, incorrect, or missing, feel free to help improve it. ### How to contribute 1. Fork the documentation repository: 2. Create a new branch for your changes. 3. Make your changes and commit them. 4. Open a Pull Request against the `main` branch with a short description of what you changed and why. ### What you can contribute * Fix typos or incorrect information * Add missing documentation for existing features * Improve existing explanations or examples * Add examples for additional languages or use cases * Add or update client profile information