pax_global_header00006660000000000000000000000064144014313100014501gustar00rootroot0000000000000052 comment=340eec5f119312cdf80950a2adc7196b26830a2e libmemcached-1.1.4/000077500000000000000000000000001440143131000141015ustar00rootroot00000000000000libmemcached-1.1.4/.builds/000077500000000000000000000000001440143131000154415ustar00rootroot00000000000000libmemcached-1.1.4/.builds/alpine.yml000066400000000000000000000013641440143131000174400ustar00rootroot00000000000000image: alpine/latest packages: - bison - cmake - flex - g++ - libatomic - libevent-dev - libstdc++ - memcached sources: - https://github.com/awesomized/libmemcached secrets: - a223d068-8d3f-4bab-a623-ed6e2887820a - d7dfe587-b433-481b-8725-d7ccd82e59fb environment: ENABLE_HASH_HSIEH: "ON" ENABLE_MEMASLAP: "ON" VERBOSE: "ON" tasks: - prepare: | ./libmemcached/.builds/scripts/prepare - configure: | maybe cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -S libmemcached -B debug - build: | maybe cmake --build debug -j2 - test: | maybe cmake --build debug -j2 --target test - install: | maybe cmake --install debug --prefix /tmp - success: | notify-gitter success libmemcached-1.1.4/.builds/freebsd.yml000066400000000000000000000024701440143131000176010ustar00rootroot00000000000000image: freebsd/latest packages: - autotools - bison - cmake - curl - cyrus-sasl - flex - libevent - pkgconf - py39-sphinx - py39-m2r - rsync - tbb sources: - https://github.com/awesomized/libmemcached secrets: - a223d068-8d3f-4bab-a623-ed6e2887820a - d7dfe587-b433-481b-8725-d7ccd82e59fb environment: ENABLE_HASH_HSIEH: "ON" ENABLE_MEMASLAP: "ON" ENABLE_SASL: "ON" VERBOSE: "ON" MEMCACHED_BINARY: "/home/build/memcached/work/stage/home/build/bin/memcached" tasks: - prepare: | ./libmemcached/.builds/scripts/prepare - memcached: | ln -s /usr/ports/Mk . ln -s /usr/ports/Templates . cp -R /usr/ports/databases/memcached . cd memcached echo bin/memcached > pkg-plist maybe make all install PREFIX=/home/build \ INSTALL_AS_USER=1 NO_PKG_REGISTER=1 PKG_REGISTER=/usr/bin/true \ SASLPWDB_CONFIGURE_ENABLE=sasl-pwdb \ OPTIONS_SET="SASL SASLPWDB" \ OPTIONS_DEFINE="SASL SASLPWDB" - configure: | maybe cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -S libmemcached -B debug - build: | maybe cmake --build debug -j2 - test: | maybe cmake --build debug -j2 --target test - install: | maybe cmake --install debug --prefix /tmp - success: | notify-gitter success libmemcached-1.1.4/.builds/openbsd.yml000066400000000000000000000015011440143131000176130ustar00rootroot00000000000000image: openbsd/latest packages: - bison - cmake - curl-- - libevent - memcached-- - pkgconf - pigz - py3-sphinx - py3-m2r - rsync-- - sudo-- - tbb sources: - https://github.com/awesomized/libmemcached secrets: - a223d068-8d3f-4bab-a623-ed6e2887820a - d7dfe587-b433-481b-8725-d7ccd82e59fb environment: ENABLE_HASH_HSIEH: "ON" ENABLE_MEMASLAP: "OFF" VERBOSE: "ON" MEMCACHED_BINARY: "memcached" tasks: - prepare: | ./libmemcached/.builds/scripts/prepare - configure: | maybe cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -S libmemcached -B debug - build: | maybe cmake --build debug -j2 - test: | maybe cmake --build debug -j2 --target test - install: | maybe cmake --install debug --prefix /tmp - success: notify-gitter success libmemcached-1.1.4/.builds/scripts/000077500000000000000000000000001440143131000171305ustar00rootroot00000000000000libmemcached-1.1.4/.builds/scripts/maybe000077500000000000000000000001441440143131000201520ustar00rootroot00000000000000#!/usr/bin/env bash "$@" ERROR=$? if test $ERROR -ne 0 then notify-gitter failure exit $ERROR fi libmemcached-1.1.4/.builds/scripts/notify-gitter000077500000000000000000000015021440143131000216600ustar00rootroot00000000000000#!/usr/bin/env bash set -eu test -f ~/.gitter || exit 0 GITTER=$(cat ~/.gitter) STATUS=$1 REPO=awesomized/libmemcached REF=$(GIT_DIR=libmemcached/.git git describe --abbrev --always) REF_URL=https://github.com/${REPO}/commits/${REF} BUILD_URL=${JOB_URL} BUILD_TAG=$(uname -o 2>/dev/null || uname -s) if expr "$BUILD_TAG" : ".*Linux" >/dev/null then BUILD_TAG=$(source /etc/os-release; echo $PRETTY_NAME) fi BUILD_CXX=$(c++ --version | head -1) if expr "${BUILD_CXX}" : "${BUILD_TAG}" >/dev/null then BUILD_ENV="${BUILD_CXX}" else BUILD_ENV="${BUILD_TAG}/${BUILD_CXX}" fi case "$STATUS" in success) LEVEL=info ;; *) LEVEL=error ;; esac MESSAGE="Sourcehut [${REPO}](${REF_URL}) (${REF}) [${STATUS}](${BUILD_URL}) (${BUILD_ENV})" curl -sS "${GITTER}" --data-urlencode "level=${LEVEL}" --data-urlencode "message=${MESSAGE}" libmemcached-1.1.4/.builds/scripts/prepare000077500000000000000000000001251440143131000205120ustar00rootroot00000000000000#!/usr/bin/env bash echo 'PATH="$PATH:~/libmemcached/.builds/scripts"' >>~/.buildenv libmemcached-1.1.4/.clang-format000066400000000000000000000072021440143131000164550ustar00rootroot00000000000000AccessModifierOffset: -2 AlignAfterOpenBracket: Align AlignConsecutiveAssignments: false #AlignConsecutiveBitFields: false (in v11) AlignConsecutiveDeclarations: false AlignConsecutiveMacros: true AlignEscapedNewlines: DontAlign # AlignOperands: AlignAfterOperator (in v11) AlignOperands: false AlignTrailingComments: true AllowAllArgumentsOnNextLine: true AllowAllConstructorInitializersOnNextLine: true AllowAllParametersOfDeclarationOnNextLine: true AllowShortBlocksOnASingleLine: Never AllowShortCaseLabelsOnASingleLine: false #AllowShortEnumsOnASingleLine: true (in v11) AllowShortFunctionsOnASingleLine: Empty AllowShortIfStatementsOnASingleLine: Never AllowShortLambdasOnASingleLine: All AllowShortLoopsOnASingleLine: true AlwaysBreakAfterReturnType: None AlwaysBreakBeforeMultilineStrings: false AlwaysBreakTemplateDeclarations: Yes BinPackArguments: true BinPackParameters: true #BitFieldColonSpacing: None (in v12) BraceWrapping: AfterCaseLabel: false AfterClass: false AfterControlStatement: MultiLine AfterEnum: false AfterExternBlock: false AfterFunction: false AfterNamespace: false AfterObjCDeclaration: false AfterStruct: false AfterUnion: false BeforeCatch: false BeforeElse: false # BeforeLambdaBody: false (in v11) # BeforeWhile: false (in v11) IndentBraces: false SplitEmptyFunction: true SplitEmptyNamespace: true SplitEmptyRecord: true BreakBeforeBinaryOperators: NonAssignment BreakBeforeBraces: Custom BreakBeforeInheritanceComma: true BreakBeforeTernaryOperators: true BreakConstructorInitializers: BeforeComma BreakConstructorInitializersBeforeComma: true BreakInheritanceList: BeforeComma BreakStringLiterals: true ColumnLimit: 100 #CommentPragmas: '^ ...' CompactNamespaces: false ConstructorInitializerAllOnOneLineOrOnePerLine: false ConstructorInitializerIndentWidth: 0 ContinuationIndentWidth: 4 Cpp11BracedListStyle: true DeriveLineEnding: true DerivePointerAlignment: false DisableFormat: false #ExperimentalAutoDetectBinPacking: false FixNamespaceComments: true ForEachMacros: - foreach - BOOST_FOREACH IncludeBlocks: Preserve IncludeCategories: - Regex: '^"lib(hashkit|(memcached(|protocol|util)))(-.*)?/' Priority: 3 - Regex: '^(<|")test/)' Priority: 2 - Regex: '.*' Priority: 1 IncludeIsMainSourceRegex: '' IndentCaseLabels: false IndentGotoLabels: false #IndentExternBlock: false (in v11) IndentPPDirectives: AfterHash IndentWidth: 2 IndentWrappedFunctionNames: false #InsertTrailingCommas: Wrapped (in v11) KeepEmptyLinesAtTheStartOfBlocks: false Language: Cpp MacroBlockBegin: '' MacroBlockEnd: '' MaxEmptyLinesToKeep: 1 NamespaceIndentation: None #PenaltyBreakAssignment: 2 #PenaltyBreakBeforeFirstCallParameter: 19 #PenaltyBreakComment: 300 #PenaltyBreakFirstLessLess: 120 #PenaltyBreakString: 1000 #PenaltyBreakTemplateDeclaration: 10 #PenaltyExcessCharacter: 1000000 #PenaltyReturnTypeOnItsOwnLine: 60 PointerAlignment: Right ReflowComments: true SortIncludes: false SortUsingDeclarations: true SpaceAfterCStyleCast: true SpaceAfterLogicalNot: false SpaceAfterTemplateKeyword: false #SpaceAroundPointerQualifiers: Both (in v12) SpaceBeforeAssignmentOperators: true SpaceBeforeCpp11BracedList: false SpaceBeforeCtorInitializerColon: true SpaceBeforeInheritanceColon: true SpaceBeforeParens: ControlStatements SpaceBeforeRangeBasedForLoopColon: true SpaceBeforeSquareBrackets: false SpaceInEmptyBlock: false SpaceInEmptyParentheses: false SpacesBeforeTrailingComments: 1 SpacesInAngles: false SpacesInConditionalStatement: false SpacesInContainerLiterals: false SpacesInCStyleCastParentheses: false SpacesInParentheses: false SpacesInSquareBrackets: false Standard: Latest #StatementMacros: # - TabWidth: 2 UseCRLF: false UseTab: Never libmemcached-1.1.4/.cppcheck/000077500000000000000000000000001440143131000157375ustar00rootroot00000000000000libmemcached-1.1.4/.cppcheck/.gitignore000066400000000000000000000000161440143131000177240ustar00rootroot00000000000000* !.gitignore libmemcached-1.1.4/.editorconfig000066400000000000000000000007141440143131000165600ustar00rootroot00000000000000root = true [*] charset = utf-8 end_of_line = lf insert_final_newline = true trim_trailing_whitespace = true [*.{C,c,cc,cpp,cxx,H,h,hh,hpp,hxx}] indent_style = space indent_size = 2 [*.{yml,yaml}] indent_style = space indent_size = 2 [*.{rst,md}] trim_trailing_whitespace = false indent_style = space indent_size = 4 [{CMakeLists.txt,CmakeConfig.txt,CMakeVersions.txt,*.cmake}] indent_style = space indent_size = 4 [{Makefile,*.mk}] indent_style = tab libmemcached-1.1.4/.gitattributes000066400000000000000000000001511440143131000167710ustar00rootroot00000000000000/ChangeLog* merge=touch /docs/sources/ChangeLog* merge=touch /docs/sources/issues.rst merge=touch libmemcached-1.1.4/.github/000077500000000000000000000000001440143131000154415ustar00rootroot00000000000000libmemcached-1.1.4/.github/notify-gitter.sh000077500000000000000000000010221440143131000205770ustar00rootroot00000000000000#!/usr/bin/env bash REPO=${GITHUB_REPOSITORY} REF=$(basename ${GITHUB_REF}) REF_URL=https://github.com/${REPO}/commits/${REF} BUILD_URL=https://github.com/${REPO}/actions/runs/${GITHUB_RUN_ID} BUILD_ENV=${ImageOS}/${CC:-${CC_VND}-${CC_VER}} case "$1" in 1|true|success) level=info status=success ;; *) level=error status=failure ;; esac message="Github [${REPO}](${REF_URL}) (${REF}) [${status}](${BUILD_URL}) (${BUILD_ENV})" curl -sS "${GITTER}" --data-urlencode "level=${level}" --data-urlencode "message=${message}" libmemcached-1.1.4/.github/workflows/000077500000000000000000000000001440143131000174765ustar00rootroot00000000000000libmemcached-1.1.4/.github/workflows/cmake-build-ci.gen000077500000000000000000000323001440143131000227400ustar00rootroot00000000000000#!/usr/bin/env php "Linux", "Linux" => "ubuntu-22.04", "ubuntu-22.04" => "gnu", "Windows" => "windows-2022", "windows-2022" => "msvc", "macOS" => "macos-12", "macos-12" => "clang", "gnu" => [ "CC" => "gcc", "CXX" => "g++", ], "clang" => [ "CC" => "clang", "CXX" => "clang++", ], "msvc" => [ "CMAKE_GENERATOR" => "Visual Studio" ], "mingw" => [ "CMAKE_GENERATOR" => "MinGW Makefiles" ] ]; const MAP = [ "Linux" => [ 'env.OS_VER' => [ "ubuntu-22.04" => [ 'env.CC_VND' => [ "gnu" => [ 'env.CC_VER' => [ "new" => "-12", "cur" => "-11", "old" => "-10", ] ], "clang" => [ 'env.CC_VER' => [ "new" => "-14", "cur" => "-13", ] ] ] ], "ubuntu-20.04" => [ 'env.CC_VND' => [ "gnu" => [ 'env.CC_VER' => [ "new" => "-10", "cur" => "-9", ] ], "clang" => [ 'env.CC_VER' => [ "new" => "-12", "cur" => "-11", "old" => "-10" ] ] ] ], ], ], "Windows" => [ 'env.OS_VER' => [ "windows-2022" => [ 'env.CC_VND' => [ "msvc" => [ "env.CC_VER" => [ "cur" => " 17 2022" ] ], "mingw" => [ "env.CC_VER" => [ "cur" => "" ] ] ] ], "windows-2019" => [ 'env.CC_VND' => [ "msvc" => [ "env.CC_VER" => [ "cur" => " 16 2019" ] ], "mingw" => [ "env.CC_VER" => [ "cur" => "" ] ] ] ], ] ], "macOS" => [] ]; const ENV = [ "ubuntu-22.04" => [ "clang" => [ "new" => [ "CXXFLAGS" => "-stdlib=libc++", "INSTALL_CXX" => "libc++-14-dev libc++abi-14-dev" ], "cur" => [ "CXXFLAGS" => "-stdlib=libc++", "INSTALL_CXX" => "libc++-13-dev libc++abi-13-dev" ], ] ], "ubuntu-20.04" => [ "clang" => [ "new" => [ "CXXFLAGS" => "-stdlib=libc++", "INSTALL_CXX" => "libc++-12-dev libc++abi-12-dev" ], "cur" => [ "CXXFLAGS" => "-stdlib=libc++", "INSTALL_CXX" => "libc++-11-dev libc++abi-11-dev" ], "old" => [ "CXXFLAGS" => "-stdlib=libc++", "INSTALL_CXX" => "libc++-10-dev libc++abi-10-dev" ], ] ], ]; function set_addpath($os_vnd, $paths) { foreach ((array) $paths as $path) if ($os_vnd == "Windows") { ?> echo "" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append echo 'PATH="$PATH:"' >> ${GITHUB_ENV} echo '=' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append echo ="" >> ${GITHUB_ENV} - name: Prepare environment (Windows) run: | echo "c:\msys64\usr\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append echo "c:\msys64\mingw64\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append $os_vers) { foreach ($os_vers as $os_ver => $cc_vnds_) { if (!$splat_map && DEF[DEF["os"]] != $os_ver) continue; foreach ($cc_vnds_ as $cc_vnd_is => $cc_vnds) { foreach ($cc_vnds as $cc_vnd => $cc_vers_) { if (!$splat_map && DEF[DEF[DEF["os"]]] != $cc_vnd) continue; foreach ($cc_vers_ as $cc_ver_is => $cc_vers) { foreach ($cc_vers as $cc_ver => $ver) { if (!$splat_map && "cur" != $cc_ver) continue; ?> - name: Prepare environment () if: () run: | $val) { step_setenv($os_vnd, $env, $val . $ver); } if (isset(ENV[$os_ver][$cc_vnd][$cc_ver])) { foreach (ENV[$os_ver][$cc_vnd][$cc_ver] as $env => $val) { step_setenv($os_vnd, $env, $val); } } } } } } } } } } function steps_getdeps($os_vnd) { ?> - name: Install dependencies () if: runner.os == '' run: | sudo apt-get update -y sudo apt-get install -my \ libevent-dev \ libsasl2-dev \ libtbb-dev \ python3-sphinx \ ${INSTALL_MEMCACHED} \ ${INSTALL_CC} ${INSTALL_CXX} sudo systemctl stop memcached || true run: | brew install bison flex libevent pkg-config sphinx-doc ${INSTALL_MEMCACHED} brew services stop memcached || true echo MEMCACHED_BINARY="/usr/local/bin/memcached" >> ${GITHUB_ENV} uses: msys2/setup-msys2@v2 with: release: false path-type: inherit install: >- bison flex rsync openssh - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} OS_VND: # OS_VER: # CC_VND: # CC_VER: cur name: cmake-build-ci on: workflow_dispatch: release: types: [published] push: paths: - ".github/workflows/cmake-build-ci*" - "CMake*" - "CPack*" - "contrib/**" - "include/**" - "src/**" - "test/**" branches-ignore: - gh-pages pull_request: branches: - master - v1.x env: # defaults INSTALL_MEMCACHED: memcached ENABLE_SASL: "OFF" # ^ almost no memcached distribution package has built in sasl support ENABLE_HASH_HSIEH: "ON" ENABLE_DTRACE: "OFF" OS_VND: # OS_VER: # CC_VND: # CC_VER: cur jobs: # sanitizer debug-sanitizer: name: debug sanitizer runs-on: # strategy: matrix: sanitizer: ['address;undefined', 'thread'] env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" ENABLE_SANITIZERS: ${{ matrix.sanitizer }} steps: - uses: actions/checkout@v3 # memcached debug-memcached: name: debug memcached runs-on: # continue-on-error: true strategy: matrix: memcached: ['master', '1.6.17', '1.6.9', '1.5.22', '1.5.6'] env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" INSTALL_MEMCACHED: "" MEMCACHED_PREFIX: "/tmp" ENABLE_SASL: "ON" steps: - uses: actions/checkout@v3 - uses: actions/checkout@v3 with: repository: memcached/memcached path: memcached ref: ${{ matrix.memcached }} # coverage linux debug-coverage-linux: name: debug coverage () runs-on: # env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" CFLAGS: "-O0 --coverage" CXXFLAGS: "-O0 --coverage" INSTALL_MEMCACHED: "" MEMCACHED_PREFIX: "/tmp" ENABLE_SASL: "ON" steps: - uses: actions/checkout@v3 - uses: actions/checkout@v3 with: repository: memcached/memcached path: memcached ref: 1.6.7 - uses: codecov/codecov-action@v3 # coverage mac debug-coverage-mac: name: debug coverage () runs-on: # env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" CFLAGS: "-O0 --coverage" CXXFLAGS: "-O0 --coverage" continue-on-error: true steps: - uses: actions/checkout@v3 - uses: codecov/codecov-action@v3 # mac release builds mac: name: release strategy: fail-fast: false matrix: os_ver: [macos-12, macos-11, macos-10.15] runs-on: ${{ matrix.os_ver }} env: CMAKE_BUILD_TYPE: "Release" BUILD_DOCS_MANGZ: "ON" continue-on-error: true steps: - uses: actions/checkout@v3 with: fetch-depth: 0 # windows release builds windows: name: release strategy: fail-fast: false matrix: os_ver: [windows-2022, windows-2019] cc_vnd: [msvc, mingw] cc_ver: [cur] runs-on: ${{ matrix.os_ver }} continue-on-error: true env: CMAKE_BUILD_TYPE: "Release" OS_VND: Windows OS_VER: ${{ matrix.os_ver }} CC_VND: ${{ matrix.cc_vnd }} CC_VER: ${{ matrix.cc_ver }} steps: - uses: actions/checkout@v3 with: fetch-depth: 0 # linux release builds release: strategy: fail-fast: false matrix: os_ver: [ubuntu-22.04, ubuntu-20.04] cc_vnd: [gnu, clang] cc_ver: [new, cur, old] runs-on: ${{ matrix.os_ver }} continue-on-error: ${{ matrix.cc_vnd == 'clang' }} env: CMAKE_BUILD_TYPE: "Release" BUILD_DOCS_MANGZ: "ON" OS_VND: Linux OS_VER: ${{ matrix.os_ver }} CC_VND: ${{ matrix.cc_vnd }} CC_VER: ${{ matrix.cc_ver }} steps: - uses: actions/checkout@v3 with: fetch-depth: 0 libmemcached-1.1.4/.github/workflows/cmake-build-ci.yml000066400000000000000000000656041440143131000230020ustar00rootroot00000000000000# Generated file; do not edit! name: cmake-build-ci on: workflow_dispatch: release: types: [published] push: paths: - ".github/workflows/cmake-build-ci*" - "CMake*" - "CPack*" - "contrib/**" - "include/**" - "src/**" - "test/**" branches-ignore: - gh-pages pull_request: branches: - master - v1.x env: # defaults INSTALL_MEMCACHED: memcached ENABLE_SASL: "OFF" # ^ almost no memcached distribution package has built in sasl support ENABLE_HASH_HSIEH: "ON" ENABLE_DTRACE: "OFF" OS_VND: Linux # OS_VER: ubuntu-22.04 # CC_VND: gnu # CC_VER: cur jobs: # sanitizer debug-sanitizer: name: debug sanitizer runs-on: ubuntu-22.04 # strategy: matrix: sanitizer: ['address;undefined', 'thread'] env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" ENABLE_SANITIZERS: ${{ matrix.sanitizer }} steps: - uses: actions/checkout@v3 - name: Prepare environment (for cur gnu on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='gnu') && (env.CC_VER=='cur') run: | echo CC="gcc-11" >> ${GITHUB_ENV} echo CXX="g++-11" >> ${GITHUB_ENV} - name: Install dependencies (Linux) if: runner.os == 'Linux' run: | sudo apt-get update -y sudo apt-get install -my \ libevent-dev \ libsasl2-dev \ libtbb-dev \ python3-sphinx \ ${INSTALL_MEMCACHED} \ ${INSTALL_CC} ${INSTALL_CXX} sudo systemctl stop memcached || true - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} # memcached debug-memcached: name: debug memcached runs-on: ubuntu-22.04 # continue-on-error: true strategy: matrix: memcached: ['master', '1.6.17', '1.6.9', '1.5.22', '1.5.6'] env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" INSTALL_MEMCACHED: "" MEMCACHED_PREFIX: "/tmp" ENABLE_SASL: "ON" steps: - uses: actions/checkout@v3 - uses: actions/checkout@v3 with: repository: memcached/memcached path: memcached ref: ${{ matrix.memcached }} - name: Prepare environment (for cur gnu on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='gnu') && (env.CC_VER=='cur') run: | echo CC="gcc-11" >> ${GITHUB_ENV} echo CXX="g++-11" >> ${GITHUB_ENV} - name: Install dependencies (Linux) if: runner.os == 'Linux' run: | sudo apt-get update -y sudo apt-get install -my \ libevent-dev \ libsasl2-dev \ libtbb-dev \ python3-sphinx \ ${INSTALL_MEMCACHED} \ ${INSTALL_CC} ${INSTALL_CXX} sudo systemctl stop memcached || true - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} # coverage linux debug-coverage-linux: name: debug coverage (ubuntu-22.04, gnu) runs-on: ubuntu-22.04 # env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" CFLAGS: "-O0 --coverage" CXXFLAGS: "-O0 --coverage" INSTALL_MEMCACHED: "" MEMCACHED_PREFIX: "/tmp" ENABLE_SASL: "ON" steps: - uses: actions/checkout@v3 - uses: actions/checkout@v3 with: repository: memcached/memcached path: memcached ref: 1.6.7 - name: Prepare environment (for cur gnu on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='gnu') && (env.CC_VER=='cur') run: | echo CC="gcc-11" >> ${GITHUB_ENV} echo CXX="g++-11" >> ${GITHUB_ENV} - name: Install dependencies (Linux) if: runner.os == 'Linux' run: | sudo apt-get update -y sudo apt-get install -my \ libevent-dev \ libsasl2-dev \ libtbb-dev \ python3-sphinx \ ${INSTALL_MEMCACHED} \ ${INSTALL_CC} ${INSTALL_CXX} sudo systemctl stop memcached || true - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} - uses: codecov/codecov-action@v3 # coverage mac debug-coverage-mac: name: debug coverage (macos-12, clang) runs-on: macos-12 # env: CMAKE_BUILD_TYPE: "Debug" BUILD_TESTING: "ON" VERBOSE: "ON" CFLAGS: "-O0 --coverage" CXXFLAGS: "-O0 --coverage" OS_VND: macOS # OS_VER: macos-12 # CC_VND: clang # CC_VER: cur continue-on-error: true steps: - uses: actions/checkout@v3 - name: Install dependencies (macOS) if: runner.os == 'macOS' run: | brew install bison flex libevent pkg-config sphinx-doc ${INSTALL_MEMCACHED} brew services stop memcached || true echo MEMCACHED_BINARY="/usr/local/bin/memcached" >> ${GITHUB_ENV} - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} - uses: codecov/codecov-action@v3 # mac release builds mac: name: release strategy: fail-fast: false matrix: os_ver: [macos-12, macos-11, macos-10.15] runs-on: ${{ matrix.os_ver }} env: CMAKE_BUILD_TYPE: "Release" BUILD_DOCS_MANGZ: "ON" OS_VND: macOS # OS_VER: macos-12 # CC_VND: clang # CC_VER: cur continue-on-error: true steps: - uses: actions/checkout@v3 with: fetch-depth: 0 - name: Install dependencies (macOS) if: runner.os == 'macOS' run: | brew install bison flex libevent pkg-config sphinx-doc ${INSTALL_MEMCACHED} brew services stop memcached || true echo MEMCACHED_BINARY="/usr/local/bin/memcached" >> ${GITHUB_ENV} - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} # windows release builds windows: name: release strategy: fail-fast: false matrix: os_ver: [windows-2022, windows-2019] cc_vnd: [msvc, mingw] cc_ver: [cur] runs-on: ${{ matrix.os_ver }} continue-on-error: true env: CMAKE_BUILD_TYPE: "Release" OS_VND: Windows OS_VER: ${{ matrix.os_ver }} CC_VND: ${{ matrix.cc_vnd }} CC_VER: ${{ matrix.cc_ver }} steps: - uses: actions/checkout@v3 with: fetch-depth: 0 - name: Prepare environment (Windows) run: | echo "c:\msys64\usr\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append echo "c:\msys64\mingw64\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append - name: Prepare environment (for cur msvc on windows-2022) if: (env.OS_VER=='windows-2022') && (env.CC_VND=='msvc') && (env.CC_VER=='cur') run: | echo 'CMAKE_GENERATOR=Visual Studio 17 2022' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append - name: Prepare environment (for cur mingw on windows-2022) if: (env.OS_VER=='windows-2022') && (env.CC_VND=='mingw') && (env.CC_VER=='cur') run: | echo 'CMAKE_GENERATOR=MinGW Makefiles' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append - name: Prepare environment (for cur msvc on windows-2019) if: (env.OS_VER=='windows-2019') && (env.CC_VND=='msvc') && (env.CC_VER=='cur') run: | echo 'CMAKE_GENERATOR=Visual Studio 16 2019' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append - name: Prepare environment (for cur mingw on windows-2019) if: (env.OS_VER=='windows-2019') && (env.CC_VND=='mingw') && (env.CC_VER=='cur') run: | echo 'CMAKE_GENERATOR=MinGW Makefiles' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append - name: Install dependencies (Windows) if: runner.os == 'Windows' uses: msys2/setup-msys2@v2 with: release: false path-type: inherit install: >- bison flex rsync openssh - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} # linux release builds release: strategy: fail-fast: false matrix: os_ver: [ubuntu-22.04, ubuntu-20.04] cc_vnd: [gnu, clang] cc_ver: [new, cur, old] runs-on: ${{ matrix.os_ver }} continue-on-error: ${{ matrix.cc_vnd == 'clang' }} env: CMAKE_BUILD_TYPE: "Release" BUILD_DOCS_MANGZ: "ON" OS_VND: Linux OS_VER: ${{ matrix.os_ver }} CC_VND: ${{ matrix.cc_vnd }} CC_VER: ${{ matrix.cc_ver }} steps: - uses: actions/checkout@v3 with: fetch-depth: 0 - name: Prepare environment (for new gnu on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='gnu') && (env.CC_VER=='new') run: | echo CC="gcc-12" >> ${GITHUB_ENV} echo CXX="g++-12" >> ${GITHUB_ENV} - name: Prepare environment (for cur gnu on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='gnu') && (env.CC_VER=='cur') run: | echo CC="gcc-11" >> ${GITHUB_ENV} echo CXX="g++-11" >> ${GITHUB_ENV} - name: Prepare environment (for old gnu on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='gnu') && (env.CC_VER=='old') run: | echo CC="gcc-10" >> ${GITHUB_ENV} echo CXX="g++-10" >> ${GITHUB_ENV} - name: Prepare environment (for new clang on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='clang') && (env.CC_VER=='new') run: | echo CC="clang-14" >> ${GITHUB_ENV} echo CXX="clang++-14" >> ${GITHUB_ENV} echo CXXFLAGS="-stdlib=libc++" >> ${GITHUB_ENV} echo INSTALL_CXX="libc++-14-dev libc++abi-14-dev" >> ${GITHUB_ENV} - name: Prepare environment (for cur clang on ubuntu-22.04) if: (env.OS_VER=='ubuntu-22.04') && (env.CC_VND=='clang') && (env.CC_VER=='cur') run: | echo CC="clang-13" >> ${GITHUB_ENV} echo CXX="clang++-13" >> ${GITHUB_ENV} echo CXXFLAGS="-stdlib=libc++" >> ${GITHUB_ENV} echo INSTALL_CXX="libc++-13-dev libc++abi-13-dev" >> ${GITHUB_ENV} - name: Prepare environment (for new gnu on ubuntu-20.04) if: (env.OS_VER=='ubuntu-20.04') && (env.CC_VND=='gnu') && (env.CC_VER=='new') run: | echo CC="gcc-10" >> ${GITHUB_ENV} echo CXX="g++-10" >> ${GITHUB_ENV} - name: Prepare environment (for cur gnu on ubuntu-20.04) if: (env.OS_VER=='ubuntu-20.04') && (env.CC_VND=='gnu') && (env.CC_VER=='cur') run: | echo CC="gcc-9" >> ${GITHUB_ENV} echo CXX="g++-9" >> ${GITHUB_ENV} - name: Prepare environment (for new clang on ubuntu-20.04) if: (env.OS_VER=='ubuntu-20.04') && (env.CC_VND=='clang') && (env.CC_VER=='new') run: | echo CC="clang-12" >> ${GITHUB_ENV} echo CXX="clang++-12" >> ${GITHUB_ENV} echo CXXFLAGS="-stdlib=libc++" >> ${GITHUB_ENV} echo INSTALL_CXX="libc++-12-dev libc++abi-12-dev" >> ${GITHUB_ENV} - name: Prepare environment (for cur clang on ubuntu-20.04) if: (env.OS_VER=='ubuntu-20.04') && (env.CC_VND=='clang') && (env.CC_VER=='cur') run: | echo CC="clang-11" >> ${GITHUB_ENV} echo CXX="clang++-11" >> ${GITHUB_ENV} echo CXXFLAGS="-stdlib=libc++" >> ${GITHUB_ENV} echo INSTALL_CXX="libc++-11-dev libc++abi-11-dev" >> ${GITHUB_ENV} - name: Prepare environment (for old clang on ubuntu-20.04) if: (env.OS_VER=='ubuntu-20.04') && (env.CC_VND=='clang') && (env.CC_VER=='old') run: | echo CC="clang-10" >> ${GITHUB_ENV} echo CXX="clang++-10" >> ${GITHUB_ENV} echo CXXFLAGS="-stdlib=libc++" >> ${GITHUB_ENV} echo INSTALL_CXX="libc++-10-dev libc++abi-10-dev" >> ${GITHUB_ENV} - name: Install dependencies (Linux) if: runner.os == 'Linux' run: | sudo apt-get update -y sudo apt-get install -my \ libevent-dev \ libsasl2-dev \ libtbb-dev \ python3-sphinx \ ${INSTALL_MEMCACHED} \ ${INSTALL_CC} ${INSTALL_CXX} sudo systemctl stop memcached || true - name: Build memcached if: runner.os != 'Windows' run: | if test -d memcached then cd memcached ./autogen.sh cp configure{,.old} && sed -e 's/-Werror//g' configure ./configure CFLAGS="-O2 -pipe -fcommon" \ --prefix=${MEMCACHED_PREFIX} \ --enable-sasl \ --enable-sasl-pwdb \ --disable-coverage \ --disable-dependency-tracking \ --disable-docs \ --disable-extstore \ --disable-option-checking \ ; make -j2 make install cd .. echo MEMCACHED_BINARY="${MEMCACHED_PREFIX}/bin/memcached" >> ${GITHUB_ENV} fi - name: Generate build tree (${{ env.CMAKE_BUILD_TYPE }}) run: cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build - name: Build all with ${{ env.CXX }} ${{ env.CXXFLAGS }} run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 - name: Test if: env.BUILD_TESTING == 'ON' run: cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target test - name: Install if: env.BUILD_TESTING == 'ON' run: cmake --install build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /tmp - name: Failed tests log if: ${{ env.BUILD_TESTING == 'ON' && failure() }} run: cat build/Testing/Temporary/LastTest.log || true - name: Package env: PUSH_ARTIFACTS_ID: ${{ secrets.PUSH_ARTIFACTS_ID }} if: env.PUSH_ARTIFACTS_ID != '' && env.CMAKE_BUILD_TYPE == 'Release' run: | cmake -DCMAKE_BUILD_TYPE=${{ env.CMAKE_BUILD_TYPE }} -S . -B build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake -DCPACK_COMPONENT_INSTALL=ON build cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target package cmake --build build --config ${{ env.CMAKE_BUILD_TYPE }} -j2 --target push-artifacts - name: Notify Gitter env: GITTER: ${{ secrets.GITTER }} if: (success() || failure()) && env.GITTER != '' run: bash .github/notify-gitter.sh ${{ job.status }} libmemcached-1.1.4/.github/workflows/docs-publish-pages.yml000066400000000000000000000016361440143131000237200ustar00rootroot00000000000000name: docs-publish-pages on: workflow_dispatch: release: types: [published] push: paths: - 'docs/**' - 'ChangeLog*' branches: - v1.x env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} jobs: publish: runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 - name: Install dependencies run: | sudo apt-get update -y sudo apt-get install -y \ python3-m2r \ python3-sphinx \ python3-sphinx-rtd-theme - name: Configure run: | cmake -DBUILD_DOCSONLY=ON -DBUILD_DOCS_HTML=ON -DBUILD_DOCS_MAN=OFF -S . -B build - name: Build run: | make -C build html - uses: crazy-max/ghaction-github-pages@v2 if: success() with: jekyll: false keep_history: true allow_empty_commit: false build_dir: build/docs/html libmemcached-1.1.4/.gitignore000066400000000000000000000002711440143131000160710ustar00rootroot00000000000000*~ *.log *.orig *.output *.pop *.rej *.bak *TAGS *patch .cproject .idea/ .project .settings/ cmake-build-*/ ccmake-*/ tags venv/ /infer-out/ /docs/gh-pages/pages/ /docs/gh-pages/build/ libmemcached-1.1.4/AUTHORS000066400000000000000000000015741440143131000151600ustar00rootroot00000000000000Andre Cruz - Help with getting the CRC Hash function to match other connectors Brian Aker - Original work, Client Library, Tools Brian Pontz - Hsieh hash Cal Heldenbrand - Awesome feedback on performance Dustin Sallings - Insight into protocol Eirik A. Nygaard - IO Patch Eric Lambert - UDP work Kevin Dalley - Bug Fixes Marcelo Fernandez - TCP/IP timeout pieces Mark Atwood - Tools, Docs Michael Wallner - CMake build, Catch2 tests, Resurrection Mingqiang Zhuang - Rewrite of memslap Monty Taylor - Debian Packages, Cleanup work for configure, Build Releated (Pandora) Padraig O'Sullivan - C++ Interface Patrick Galbraith - work on C++ interface Ross McFarland - Idea for sorting servers Tim Bunce - Docs, Perl Driver work and feedback on API Tobias Luetke - Performance Feedback Toru Maesaka - Stats analysis Trond Norbye - Binary protocol, Misc Yin Chen - Ketama support/weighted support libmemcached-1.1.4/BUGS.md000066400000000000000000000036431440143131000151710ustar00rootroot00000000000000# Bugs, Known Issues and Insufficiencies ## libhashkit libhashkit is not usable for general purpose hashing, because it is geared to usage by libmemcached. ### MurMur Hashkit's MurMur/MurMur3 are limited to the lower 32 bits. ### crc32 Commit "[More Hashing methods](https://github.com/awesomized/libmemcached/commits/1207354f)" from October 2007 first released in v0.8, which main intention seems to have been to add FNV1 hash algos, changed the result of the crc32 hash to only its upper 16 bits sans MSB, without any additional comment. The implementations referred to in the file header (Postgres and BSD) do not exhibit this behavior. A [bug report](https://bugs.launchpad.net/libmemcached/+bug/604178) was filed three years later on launchpad, which was marked `Won't fix` with the comment that it was for compatibility with other "drivers", which supposedly refers to other memcached client libraries. ## libmemcached ### Replication This is a somewhat badly named feature, because it **does not** provide any of the guaranties one would expect from a proper replication. One can set the intended number of additional servers where data should be stored with the behavior `MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS` and specify whether `MGET`s/`GET`s should read from a random server with `MEMCACHED_BEHAVIOR_RANDOMIZE_REPLICA_READ`. `DELETE`s will try to delete the key from all replicas. The binary protocol is required and any other command is unaffected. ### TLS/SSL libmemcached does not support TLS/SSL, yet. See [github issue #37](https://github.com/awesomized/libmemcached/issues/37). ### Coroutines and event loops libmemcached does not support explicit asynchronous usage, yet. See [github issue #54](https://github.com/awesomized/libmemcached/issues/54). ### META protocol libmemcached deos not support memcached's META protocol, yet. See [github issue #121](https://github.com/awesomized/libmemcached/issues/121). libmemcached-1.1.4/CMake/000077500000000000000000000000001440143131000150615ustar00rootroot00000000000000libmemcached-1.1.4/CMake/CheckAtomics.cmake000066400000000000000000000033121440143131000204170ustar00rootroot00000000000000# sets HAVE_ATOMICS, checks for : # - C++11 std::atomic: HAVE_CXX_STDATOMIC # - C11 stdatomic: HAVE_C_STDATOMIC # - builtin __atomic: HAVE_BUILTIN_ATOMIC # - builtin __sync: HAVE_BUILTIN_SYNC # - atomic_add_nv: HAVE_ATOMIC_ADD_NV configure_define(HAVE_ATOMICS) check_cxx_source(" #include int main() { std::atomic i(0); return atomic_load(&i); }" HAVE_CXX_STDATOMIC) check_c_source(" #include int main() { atomic_int i; atomic_init(&i, 0); return atomic_load(&i); }" HAVE_C_STDATOMIC) configure_define(HAVE_BUILTIN_ATOMIC) configure_define_literal(BUILTIN_ATOMIC_PREFIX) foreach(BUILTIN_ATOMIC_PREFIX IN ITEMS __c11 _ "") check_c_source(" int main() { long l = -1; return ${BUILTIN_ATOMIC_PREFIX}_atomic_add_fetch(&l,1,__ATOMIC_RELAXED); }" HAVE_BUILTIN_ATOMIC${BUILTIN_ATOMIC_PREFIX}) if (HAVE_BUILTIN_ATOMIC${BUILTIN_ATOMIC_PREFIX}) set(HAVE_BUILTIN_ATOMIC 1) break() endif() endforeach() check_c_source(" int main() { long l = -1; return __sync_add_and_fetch(&l,1); }" HAVE_BUILTIN_SYNC) check_c_source(" #include int main() { volatile uint_t i = 0; return atomic_add_int_nv(&i, 1) == 1 ? 0 : -1; }" HAVE_ATOMIC_ADD_NV) if ( (HAVE_CXX_STDATOMIC) OR (HAVE_C_STDATOMIC) OR (HAVE_BUILTIN_ATOMIC) OR (HAVE_BUILTIN_SYNC) OR (HAVE_ATOMIC_ADD_NV)) set(HAVE_ATOMICS 1 CACHE INTERNAL "HAVE_ATOMICS") endif() libmemcached-1.1.4/CMake/CheckBacktrace.cmake000066400000000000000000000006041440143131000207000ustar00rootroot00000000000000find_package(Backtrace) if(Backtrace_FOUND) configure_set(HAVE_BACKTRACE 1) configure_define_header(Backtrace_HEADER) set(BACKTRACE BACKTRACE) add_library(BACKTRACE INTERFACE IMPORTED) set_target_properties(BACKTRACE PROPERTIES INTERFACE_LINK_LIBRARIES "${Backtrace_LIBRARIES}" INTERFACE_INCLUDE_DIRECTORIES "${Backtrace_INCLUDE_DIR}") endif() libmemcached-1.1.4/CMake/CheckByteswap.cmake000066400000000000000000000033151440143131000206210ustar00rootroot00000000000000# defines HAVE_BYTESWAP # optionally defines BYTESWAP_HEADER # optionally defines BYTESWAP_32 # # checks whether the following compiles: # __builtin_bswap32(): defines HAVE_BUILTIN_BSWAP32 # # else checks: # byteswap.h: defines HAVE_BYTESWAP_H # bswap_32() in byteswap.h: defines HAVE_BSWAP_32 # # else checks: # sys/endian.h: defines HAVE_SYS_ENDIAN_H # bswap32() in sys/endian.h: defines HAVE_BSWAP32 # include(TestBigEndian) test_big_endian(WORDS_BIGENDIAN) configure_define(WORDS_BIGENDIAN) configure_define(HAVE_BYTESWAP) configure_define_header(BYTESWAP_HEADER) configure_define_literal(BYTESWAP_32) check_c_source(" #include int main() { uint32_t a = 1, b = __builtin_bswap32(a); return b; }" HAVE_BUILTIN_BSWAP32 ) if(HAVE_BUILTIN_BSWAP32) configure_undef(BYTESWAP_HEADER) set(BYTESWAP_32 __builtin_bswap32 CACHE INTERNAL "BYTESWAP_32") set(HAVE_BYTESWAP 1 CACHE INTERNAL "HAVE_BYTESWAP") return() endif() check_include(byteswap.h) check_symbol(bswap_32 byteswap.h) if(HAVE_BSWAP_32) if(HAVE_BYTESWAP_H) set(BYTESWAP_HEADER byteswap.h CACHE INTERNAL "BYTESWAP_HEADER") endif() set(BYTESWAP_32 bswap_32 CACHE INTERNAL "BYTESWAP_32") set(HAVE_BYTESWAP 1 CACHE INSTERNAL "HAVE_BYTESWAP") return() endif() check_include(sys/endian.h) check_symbol(bswap32 sys/endian.h) if(HAVE_BSWAP32) if(HAVE_SYS_ENDIAN_H) set(BYTESWAP_HEADER sys/endian.h CACHE INTERNAL "BYTESWAP_HEADER") endif() set(BYTESWAP_32 bswap32 CACHE INTERNAL "BYTESWAP_32") set(HAVE_BYTESWAP 1 CACHE INTERNAL "HAVE_BYTESWAP") return() endif() configure_undef(BYTESWAP_HEADER) configure_undef(BYTESWAP_32) set(HAVE_BYTESWAP 0) libmemcached-1.1.4/CMake/CheckCpp17Parallelism.cmake000066400000000000000000000015431440143131000221040ustar00rootroot00000000000000configure_define(HAVE_CPP17_PARALLELISM) # # see CheckTbb.cmake on change # check_cxx_source_compiles(" #include #include #include int main() { std::vector a = {1,2,3}; std::all_of(std::execution::par, a.begin(), a.end(), [](char i) { return i>0; }); } " HAVE_CPP17_PARALLELISM ) if(HAVE_CPP17_PARALLELISM) add_library(cpp17::parallelism INTERFACE IMPORTED) # noting to be done else() include(CheckTbb) set(HAVE_CPP17_PARALLELISM "${HAVE_TBB}") if(HAVE_TBB) add_library(cpp17::parallelism INTERFACE IMPORTED) target_link_libraries(cpp17::parallelism INTERFACE "${LIBTBB}") endif() endif() libmemcached-1.1.4/CMake/CheckDebug.cmake000066400000000000000000000045741440143131000200610ustar00rootroot00000000000000 function(set_flag FLAG DEFAULT) unset(FLAG_CONSTANT) string(MAKE_C_IDENTIFIER CXX${FLAG} FLAG_CONSTANT) check_cxx_compiler_flag(${FLAG} ${FLAG_CONSTANT}) if(${FLAG_CONSTANT}) add_compile_options(${FLAG}) elseif(DEFAULT) add_compile_options(${DEFAULT}) endif() endfunction() macro(check_sanitizer VAR NAME LIB) message(STATUS "Checking for sanitizer: ${NAME} (-l${LIB})") if(${NAME} IN_LIST ${VAR} OR ${LIB} IN_LIST ${VAR}) make_have_identifier(${LIB} HAVE) cmake_push_check_state(RESET) set(CMAKE_REQUIRED_LIBRARIES -fsanitize=${NAME}) check_cxx_compiler_flag(-fsanitize=${NAME} ${HAVE}) cmake_pop_check_state() if(${HAVE}) add_compile_definitions(${HAVE}) add_compile_options(-fsanitize=${NAME}) link_libraries(-fsanitize=${NAME}) set_flag(-fsanitize-recover=${NAME} IGNORE) message(STATUS " OK: sanitizer ${NAME}") else() message(STATUS " NO: not supported") endif() else() message(STATUS " NO: not requested") endif() endmacro() if(CMAKE_BUILD_TYPE STREQUAL "Debug" AND NOT MSVC) add_definitions(-DDEBUG=1) if(CMAKE_CXX_FLAGS MATCHES --coverage) message("-- Coverage build detected!") message("-- Skipping debug and sanitizer flag checks.") else() set_flag(-Og -O0) set_flag(-ggdb -g) foreach(FLAG IN ITEMS -fno-inline -fno-omit-frame-pointer -fno-eliminate-unused-debug-types -funsafe-loop-optimizations -Wall -Wextra -Wdouble-promotion -Wduplicated-cond -Wduplicated-branches -Wformat=2 -Wlogical-op -Wnull-dereference -Wrestrict -Wshadow -Wunknown-pragmas -Wunsafe-loop-optimizations ) set_flag(${FLAG} IGNORE) endforeach() if(ENABLE_SANITIZERS) check_sanitizer(ENABLE_SANITIZERS address asan) check_sanitizer(ENABLE_SANITIZERS undefined ubsan) check_sanitizer(ENABLE_SANITIZERS thread tsan) check_sanitizer(ENABLE_SANITIZERS leak lsan) endif() endif() else() add_definitions(-DDEBUG=0) endif() libmemcached-1.1.4/CMake/CheckDependency.cmake000066400000000000000000000026611440143131000211040ustar00rootroot00000000000000find_package(PkgConfig) function(check_dependency NAME LIB) make_have_identifier(${NAME} HAVE) configure_define(${HAVE}) if(PKG_CONFIG_FOUND) pkg_check_modules(${NAME} lib${LIB}${ARGN} IMPORTED_TARGET) if(NOT ${NAME}_FOUND) pkg_check_modules(${NAME} ${LIB}${ARGN} IMPORTED_TARGET) endif() if(${NAME}_FOUND) set(${NAME} PkgConfig::${NAME} CACHE INTERNAL "${NAME} import target") set(${HAVE} 1 CACHE INTERNAL "${HAVE}") return() endif() endif() message(STATUS "Checking for library '${LIB}' ...") find_library(${NAME}_LIB NAMES ${LIB}) if(${NAME}_LIB) mark_as_advanced(${NAME}_LIB) message(STATUS " Found '${${NAME}_LIB}'") set(${NAME}_INCLUDES "") foreach(PATH IN_LIST CMAKE_PREFIX_PATHS) if(${NAME}_LIB MATCHES "^${PATH}") set(${NAME}_INCLUDES "${PATH}/include") break() endif() endforeach() add_library(Imported::${NAME} INTERFACE IMPORTED) set_target_properties(Imported::${NAME} PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${${NAME}_INCLUDES}" INTERFACE_LINK_LIBRARIES ${${NAME}_LIB}) set(${NAME} Imported::${NAME} CACHE INTERNAL "${NAME} import target") set(${HAVE} 1 CACHE INTERNAL "${HAVE}") return() endif() message(STATUS " Not found") endfunction() libmemcached-1.1.4/CMake/CheckDtrace.cmake000066400000000000000000000003751440143131000202300ustar00rootroot00000000000000include(EnableDtrace) if(ENABLE_DTRACE) find_package(DTrace) if(DTRACE_EXECUTABLE) configure_set(HAVE_DTRACE 1) else() message(WARNING "The dtrace command is required to enable dtrace/systemtap support.") endif() endif() libmemcached-1.1.4/CMake/CheckPkgconf.cmake000066400000000000000000000002661440143131000204140ustar00rootroot00000000000000if(CMAKE_HOST_SYSTEM_NAME MATCHES "BSD") find_program(PKGCONF pkgconf) if(PKGCONF) set(PKG_CONFIG_EXECUTABLE ${PKGCONF}) endif() endif() find_package(PkgConfig) libmemcached-1.1.4/CMake/CheckTbb.cmake000066400000000000000000000020231440143131000175250ustar00rootroot00000000000000 configure_define(HAVE_TBB) # TBBConfig only sets TBB_FOUND to FALSE check_dependency(LIBTBB tbb) if(HAVE_LIBTBB) cmake_push_check_state() get_property(LIBTBB_INCLUDEDIR TARGET ${LIBTBB} PROPERTY INTERFACE_INCLUDE_DIRECTORIES) get_property(LIBTBB_LIBRARIES TARGET ${LIBTBB} PROPERTY INTERFACE_LINK_LIBRARIES) set(CMAKE_REQUIRED_INCLUDES "${LIBTBB_INCLUDEDIR}") set(CMAKE_REQUIRED_LIBRARIES "${LIBTBB_LIBRARIES}") set(CMAKE_REQUIRED_FLAGS -std=c++17) check_cxx_include(execution -std=c++17) if(HAVE_EXECUTION) check_cxx_source_compiles(" #include #include #include int main() { std::vector a = {1,2,3}; std::all_of(std::execution::par, a.begin(), a.end(), [](char i) { return i>0; }); } " HAVE_TBB ) endif() cmake_pop_check_state() endif() libmemcached-1.1.4/CMake/CheckThreads.cmake000066400000000000000000000005021440143131000204100ustar00rootroot00000000000000set(THREADS_PREFER_PTHREAD_FLAG ON) set(CMAKE_THREAD_PREFER_PTHREAD ON) find_package(Threads) if(CMAKE_HAVE_PTHREAD_H) configure_define(HAVE_PTHREAD_H) set(HAVE_PTHREAD_H ${CMAKE_HAVE_PTHREAD_H} CACHE INTERNAL "FindThreads found pthread.h") elseif(CMAKE_USE_PTHREADS_INIT) check_cxx_include(pthread.h) endif() libmemcached-1.1.4/CMake/CheckVisibility.cmake000066400000000000000000000013571440143131000211560ustar00rootroot00000000000000configure_define(HAVE_VISIBILITY) check_flag(-fvisibility=hidden HAVE_VISIBILITY_FLAG) if(NOT HAVE_VISIBILITY_FLAG) check_flag(-Wl,-fvisibility=hidden HAVE_VISIBILITY_LINKER_FLAG) endif() check_c_source(" __attribute__ ((visibility (\"default\"))) int main(int argc, char **argv) { return *argv[argc-1]; }" HAVE_VISIBILITY_ATTR ) if(HAVE_VISIBILITY_ATTR AND (HAVE_VISIBILITY_FLAG OR HAVE_VISIBILITY_LINKER_FLAG)) if(HAVE_VISIBILITY_LINKER_FLAG) string(APPEND CMAKE_SHARED_LINKER_FLAGS " -Wl,-fvisibility=hidden") else() add_compile_options("-fvisibility=hidden") endif() set(HAVE_VISIBILITY 1 CACHE INTERNAL "-fvisibility and __attribute__((visibility(...)))") endif() libmemcached-1.1.4/CMake/EnableDtrace.cmake000066400000000000000000000051301440143131000203730ustar00rootroot00000000000000function(enable_dtrace_for TARGET PROBES_D PROBES_H) if(HAVE_DTRACE AND NOT CMAKE_CROSSCOMPILING) add_custom_command( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_H} MAIN_DEPENDENCY ${PROBES_D} COMMAND ${DTRACE_EXECUTABLE} ARGS -x nolibs -h -s ${CMAKE_CURRENT_SOURCE_DIR}/${PROBES_D} -o ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_H} ) target_sources(${TARGET} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_H}) if(CMAKE_HOST_SYSTEM_NAME STREQUAL "Linux") add_custom_command( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_probes.o MAIN_DEPENDENCY ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_H} COMMAND ${DTRACE_EXECUTABLE} ARGS -x nolibs -G -s ${CMAKE_CURRENT_SOURCE_DIR}/${PROBES_D} -o ${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_probes.o ) target_sources(${TARGET} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_probes.o) set_source_files_properties(${TARGET}_probes.o PROPERTIES GENERATED true EXTERNAL_OBJECT true) return() endif() cmake_host_system_information(RESULT OS_RELEASE QUERY OS_RELEASE) if(NOT (CMAKE_HOST_SYSTEM_NAME STREQUAL Darwin AND OS_RELEASE VERSION_GREATER_EQUAL 11)) set(PROBES_C ${TARGET}_probes.cc) file(GENERATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_C} CONTENT "#include \"${PROBES_H}\"\n" ) add_custom_command( TARGET ${TARGET} PRE_LINK DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_H} COMMAND rm -f ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${TARGET}.dir/${PROBES_C}.o COMMAND ${DTRACE_EXECUTABLE} -x nolibs -G -s ${CMAKE_CURRENT_SOURCE_DIR}/${PROBES_D} -o ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${TARGET}.dir/${PROBES_C}.o ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${TARGET}.dir/*.o ) target_sources(${TARGET} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/${PROBES_C}) endif() endif() endfunction() libmemcached-1.1.4/CMake/FindDTrace.cmake000066400000000000000000000004601440143131000200260ustar00rootroot00000000000000find_program(DTRACE_EXECUTABLE NAMES dtrace HINTS $ENV{DTRACE_DIR} PATH_SUFFIXES bin DOC "DTrace static probe generator" ) include(FindPackageHandleStandardArgs) find_package_handle_standard_args(DTrace DEFAULT_MSG DTRACE_EXECUTABLE) mark_as_advanced(DTRACE_EXECUTABLE) libmemcached-1.1.4/CMake/FindMemcached.cmake000066400000000000000000000005031440143131000205300ustar00rootroot00000000000000find_program(MEMCACHED_EXECUTABLE NAMES memcached HINTS $ENV{MEMCACHED_DIR} PATH_SUFFIXES bin DOC "memcached(1), Memcached daemon" ) include(FindPackageHandleStandardArgs) find_package_handle_standard_args(Memcached DEFAULT_MSG MEMCACHED_EXECUTABLE) mark_as_advanced(MEMCACHED_EXECUTABLE) libmemcached-1.1.4/CMake/FindSphinx.cmake000066400000000000000000000005061440143131000201360ustar00rootroot00000000000000find_program(SPHINX_EXECUTABLE NAMES sphinx-build sphinx-build-3 HINTS $ENV{SPHINX_DIR} PATH_SUFFIXES bin DOC "Sphinx documentation generator" ) include(FindPackageHandleStandardArgs) find_package_handle_standard_args(Sphinx DEFAULT_MSG SPHINX_EXECUTABLE) mark_as_advanced(SPHINX_EXECUTABLE) libmemcached-1.1.4/CMake/InstallPublicHeaders.cmake000066400000000000000000000040341440143131000221250ustar00rootroot00000000000000macro(install_public_headers DIRECTORY) # validate current directory string(FIND ${CMAKE_CURRENT_SOURCE_DIR} /include/ INCDIR REVERSE) string(FIND ${CMAKE_CURRENT_SOURCE_DIR} /src/ SRCDIR REVERSE) if((INCDIR GREATER_EQUAL 0) OR (SRCDIR GREATER_EQUAL 0)) if(INCDIR GREATER_EQUAL 0) math(EXPR POSITION "${INCDIR} + 9") else() math(EXPR POSITION "${SRCDIR} + 5") endif() string(SUBSTRING ${CMAKE_CURRENT_SOURCE_DIR} ${POSITION} -1 CHKDIR) if(NOT "${CHKDIR}" STREQUAL "${DIRECTORY}") message(SEND_ERROR "install_public_headers() directories do not match: '${CHKDIR}' != '${DIRECTORY}'") set(ENV{INVALID_CONFIGURATION} 1) endif() endif() string(REGEX MATCH "^[^/-]+" LIBRARY "${DIRECTORY}") # validate public interface version string(FIND "${DIRECTORY}" "-" DASH) if(DASH GREATER 0) string(SUBSTRING "${DIRECTORY}" 0 ${DASH} LIBRARY_BASE) string(TOUPPER ${LIBRARY_BASE} LIBRARY_UCASE) math(EXPR DASH "${DASH} + 1") string(SUBSTRING "${DIRECTORY}" ${DASH} -1 VERSION) if(NOT ${LIBRARY_UCASE}_VERSION_INC VERSION_EQUAL ${VERSION}) message(SEND_ERROR "${LIBRARY_BASE} public include directory version ${VERSION} != " ${${LIBRARY_UCASE}_VERSION_INC}) set(ENV{INVALID_CONFIGURATION} 1) endif() endif() # change local includes to system includes foreach(HEADER IN ITEMS ${ARGN}) if(HEADER MATCHES "^@") string(SUBSTRING ${HEADER} 1 -1 HEADER) configure_file(${HEADER}.in ${HEADER}) set(HEADER "${CMAKE_CURRENT_BINARY_DIR}/${HEADER}") else() set(HEADER "${CMAKE_CURRENT_SOURCE_DIR}/${HEADER}") endif() install(FILES ${HEADER} COMPONENT dev DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${DIRECTORY} ) list(APPEND ${LIBRARY}_includes ${HEADER}) endforeach() set(${LIBRARY}_includes "${${LIBRARY}_includes}" PARENT_SCOPE) endmacro() libmemcached-1.1.4/CMake/_Configure.cmake000066400000000000000000000020451440143131000201440ustar00rootroot00000000000000 macro(configure_init CONFIG_HEADER_FILE) set(CONFIGURE_FILE_IN ${CONFIG_HEADER_FILE}.in) file(WRITE ${CONFIGURE_FILE_IN} "#pragma once\n") set(CONFIGURE_FILE_OUT ${CONFIG_HEADER_FILE}) endmacro() macro(configure_append) file(APPEND ${CONFIGURE_FILE_IN} ${ARGN}) endmacro() macro(configure_set VAR VAL) set(${VAR} ${VAL}) configure_append("#cmakedefine ${VAR} 1\n") endmacro() macro(configure_define VAR) configure_append("#cmakedefine ${VAR} 1\n") endmacro() macro(configure_undef VAR) configure_append("#undef ${VAR}\n") endmacro() macro(configure_define_01 VAR) configure_append("#cmakedefine01 ${VAR}\n") endmacro() macro(configure_define_literal VAR) string(TOUPPER ${VAR} UPPER) configure_append("#define ${UPPER} @${VAR}@\n") endmacro() macro(configure_define_header VAR) string(TOUPPER ${VAR} UPPER) configure_append("#define ${UPPER} <@${VAR}@>\n") endmacro() macro(configure_define_string VAR) string(TOUPPER ${VAR} UPPER) configure_append("#define ${UPPER} \"@${VAR}@\"\n") endmacro() libmemcached-1.1.4/CMake/_Include.cmake000066400000000000000000000140751440143131000176140ustar00rootroot00000000000000include(_Configure) configure_init(${CMAKE_BINARY_DIR}/mem_config.h) add_compile_definitions(${GLOBAL_DEFINITIONS}) # list(TRANSFORM) requires >=3.12 string(REPLACE ";" " -D" GLOBAL_DEFINITION_FLAGS "${GLOBAL_DEFINITIONS}") set(GLOBAL_DEFINITION_FLAGS -D${GLOBAL_DEFINITION_FLAGS}) if(CMAKE_PROJECT_NAME STREQUAL PROJECT_NAME) include(CTest) endif() include(CMakePushCheckState) include(CMakePackageConfigHelpers) macro(make_have_identifier NAME ID) string(MAKE_C_IDENTIFIER ${NAME} _make_have_identifier) string(TOUPPER ${_make_have_identifier} _make_have_identifier) set(${ID} HAVE_${_make_have_identifier}) endmacro() include(CheckCCompilerFlag) macro(check_flag FLAG HAVE) configure_define(${HAVE}) check_c_compiler_flag("${FLAG}" ${HAVE}) endmacro() include(CheckCXXCompilerFlag) macro(check_cxx_flag FLAG HAVE) configure_define(${HAVE}) check_cxx_compiler_flag("${FLAG}" ${HAVE}) endmacro() include(CheckSymbolExists) macro(check_symbol NAME HEADER) make_have_identifier(${NAME} HAVE) configure_define(${HAVE}) cmake_push_check_state() if(${ARGC} GREATER 1) string(APPEND CMAKE_REQUIRED_FLAGS " ${ARGV2}") endif() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_symbol_exists(${NAME} ${HEADER} ${HAVE}) cmake_pop_check_state() endmacro() include(CheckCXXSymbolExists) macro(check_cxx_symbol NAME HEADER) make_have_identifier(${NAME} HAVE) configure_define(${HAVE}) cmake_push_check_state() if(${ARGC} GREATER 1) string(APPEND CMAKE_REQUIRED_FLAGS " ${ARGN}") endif() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_cxx_symbol_exists(${NAME} ${HEADER} ${HAVE}) cmake_pop_check_state() endmacro() include(CheckIncludeFile) macro(check_include HEADER) make_have_identifier(${HEADER} HAVE) configure_define(${HAVE}) cmake_push_check_state() if(${ARGC} GREATER 1) string(APPEND CMAKE_REQUIRED_FLAGS " ${ARGN}") endif() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_include_file(${HEADER} ${HAVE}) cmake_pop_check_state() endmacro() include(CheckIncludeFileCXX) macro(check_cxx_include HEADER) make_have_identifier(${HEADER} HAVE) configure_define(${HAVE}) cmake_push_check_state() if(${ARGC} GREATER 1) string(APPEND CMAKE_REQUIRED_FLAGS " ${ARGN}") endif() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_include_file_cxx(${HEADER} ${HAVE}) cmake_pop_check_state() endmacro() include(CheckTypeSize) macro(check_type TYPE) make_have_identifier(${TYPE} HAVE) configure_define(${HAVE}) cmake_push_check_state() if(${ARGC} GREATER 1) list(APPEND CMAKE_EXTRA_INCLUDE_FILES ${ARGN}) endif() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_type_size(${TYPE} ${HAVE}) cmake_pop_check_state() endmacro() include(CheckCSourceCompiles) macro(check_c_source SOURCE HAVE) configure_define(${HAVE}) cmake_push_check_state() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_c_source_compiles("${SOURCE}" ${HAVE}) cmake_pop_check_state() endmacro() include(CheckCXXSourceCompiles) macro(check_cxx_source SOURCE HAVE) configure_define(${HAVE}) cmake_push_check_state() list(APPEND CMAKE_REQUIRED_DEFINITIONS ${GLOBAL_DEFINITION_FLAGS}) check_cxx_source_compiles("${SOURCE}" ${HAVE}) cmake_pop_check_state() endmacro() include(CheckBacktrace) include(CheckByteswap) include(CheckDependency) include(CheckDtrace) include(CheckPkgconf) include(CheckDebug) include(CheckThreads) include(CheckVisibility) include(InstallPublicHeaders) function(pkgconfig_export VAR VAL) get_property(PREV GLOBAL PROPERTY PKGCONFIG_${VAR}) set_property(GLOBAL PROPERTY PKGCONFIG_${VAR} "${PREV} ${VAL}") endfunction() macro(pkgconfig_import VAR) get_property(PKGCONFIG_${VAR} GLOBAL PROPERTY PKGCONFIG_${VAR}) endmacro() ## sasl configure_define_01(LIBMEMCACHED_WITH_SASL_SUPPORT) if(ENABLE_SASL) check_dependency(LIBSASL sasl2) if(HAVE_LIBSASL) set(LIBMEMCACHED_WITH_SASL_SUPPORT 1) pkgconfig_export(REQUIRES libsasl2) cmake_push_check_state() set(CMAKE_REQUIRED_INCLUDES "${LIBSASL_INCLUDEDIR}") set(CMAKE_REQUIRED_LIBRARIES "${LIBSASL_LIBRARIES}") check_symbol(sasl_client_done sasl/sasl.h) cmake_pop_check_state() endif() endif() ## hashes configure_set(HAVE_FNV64_HASH ${ENABLE_HASH_FNV64}) configure_set(HAVE_MURMUR_HASH ${ENABLE_HASH_MURMUR}) configure_set(HAVE_HSIEH_HASH ${ENABLE_HASH_HSIEH}) check_include(alloca.h) check_include(arpa/inet.h) check_include(dlfcn.h) check_include(getopt.h) check_include(libgen.h) check_include(netdb.h) check_include(netinet/in.h) check_include(netinet/tcp.h) check_include(poll.h) check_include(strings.h) check_include(sys/poll.h) check_include(sys/socket.h) check_include(sys/time.h) check_include(sys/un.h) check_include(unistd.h) check_type(in_port_t netinet/in.h) check_type(pid_t sys/types.h) check_type(ssize_t sys/types.h) check_type("struct msghdr" sys/socket.h) check_type("struct timespec" time.h) check_cxx_symbol(abi::__cxa_demangle cxxabi.h) check_symbol(CLOCK_MONOTONIC time.h) check_symbol(clock_gettime time.h) check_symbol(ERESTART errno.h) check_symbol(fcntl fcntl.h) check_symbol(gettimeofday sys/time.h) check_symbol(htonll arpa/inet.h) check_symbol(index strings.h) check_symbol(MSG_DONTWAIT sys/socket.h) check_symbol(MSG_MORE sys/socket.h) check_symbol(MSG_NOSIGNAL sys/socket.h) check_symbol(SO_RCVTIMEO sys/socket.h) check_symbol(SO_SNDTIMEO sys/socket.h) check_symbol(rand stdlib.h) check_symbol(random stdlib.h) check_symbol(realpath stdlib.h) check_symbol(sendmsg sys/socket.h) check_symbol(setenv stdlib.h) check_symbol(strerror_r string.h) check_c_source(" #include int main() { char x; return *strerror_r(0, &x, 1); }" HAVE_STRERROR_R_CHAR_P ) if(WIN32) check_include(io.h) check_include(winsock2.h) check_include(ws2tcpip.h) check_symbol(htonll winsock2.h) endif() libmemcached-1.1.4/CMakeConfig.txt000066400000000000000000000070071440143131000167540ustar00rootroot00000000000000if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES) if(NOT DEFINED ENV{CMAKE_BUILD_TYPE}) set(ENV{CMAKE_BUILD_TYPE} Release) endif() set(CMAKE_BUILD_TYPE $ENV{CMAKE_BUILD_TYPE} CACHE STRING "build type (Release, Debug, ...)" FORCE) endif() set(CMAKE_INSTALL_PREFIX /usr/local CACHE PATH "install prefix") if(APPLE) set(CMAKE_INSTALL_RPATH @loader_path CACHE STRING "set relative rpath") elseif(UNIX) # FIXME set(CMAKE_INSTALL_RPATH "\$ORIGIN" CACHE STRING "set relative rpath") endif() if(${CMAKE_VERSION} VERSION_LESS "3.12") set(TARGET_NAMELINK_COMPONENT "") else() set(TARGET_NAMELINK_COMPONENT NAMELINK_COMPONENT dev DESTINATION ${CMAKE_INSTALL_LIBDIR}) endif() set(CLIENT_PREFIX mem CACHE STRING "client prefix (default mem; i.e.: memstat, memcp, memcat ...)") option(BUILD_SHARED_LIBS "whether to build shared libraries" ON) option(BUILD_TESTING "whether to enable build of the test suite" $ENV{BUILD_TESTING}) option(BUILD_DOCSONLY "build *only* documentation" $ENV{BUILD_DOCSONLY}) option(BUILD_DOCS "build documentation" ${BUILD_DOCSONLY}) option(BUILD_DOCS_HTML "build HTML docs" ${BUILD_DOCS}) option(BUILD_DOCS_MAN "build manpages" ${BUILD_DOCS}) option(BUILD_DOCS_MANGZ "gzip manpages" ${BUILD_DOCS_MAN}) if(BUILD_DOCS_MANGZ) set(BUILD_DOCS_MAN ON CACHE BOOL "forced by BUILD_DOCS_MANGZ" FORCE) endif() if(BUILD_DOCS_MAN OR BUILD_DOCS_HTML) set(BUILD_DOCS ON CACHE BOOL "forced by BUILD_DOCS_MAN OR BUILD_DOCS_HTML" FORCE) endif() set(ENABLE_SANITIZERS "$ENV{ENABLE_SANITIZERS}" CACHE STRING "sanitizers to enable (e.g. address;undefined ...)") option(ENABLE_SASL "enable SASL support" $ENV{ENABLE_SASL}) option(ENABLE_DTRACE "enable dtrace support" $ENV{ENABLE_DTRACE}) option(ENABLE_HASH_HSIEH "enable hsieh hash support" $ENV{ENABLE_HASH_HSIEH}) if(NOT DEFINED ENV{ENABLE_HASH_FNV64}) set(ENV{ENABLE_HASH_FNV64} ON) endif() option(ENABLE_HASH_FNV64 "enable fnv64 hash support" $ENV{ENABLE_HASH_FNV64}) if(NOT DEFINED ENV{ENABLE_HASH_MURMUR}) set(ENV{ENABLE_HASH_MURMUR} ON) endif() option(ENABLE_HASH_MURMUR "enable murmur hash support" $ENV{ENABLE_HASH_MURMUR}) if(NOT DEFINED ENV{ENABLE_MEMASLAP}) set(ENV{ENABLE_MEMASLAP} ON) endif() option(ENABLE_MEMASLAP "enable memaslap client" $ENV{ENABLE_MEMASLAP}) option(ENABLE_OPENSSL_CRYPTO "enable OpenSSL's libcrypto instead of bundled AES implementation" $ENV{ENABLE_OPENSSL_CRYPTO}) if(BUILD_TESTING) set(MEMCACHED_BINARY "$ENV{MEMCACHED_BINARY}" CACHE STRING "memcached binary") set(CMAKE_CTEST_ARGUMENTS "--output-on-failure") if(CMAKE_VERSION VERSION_GREATER_EQUAL 3.17) # available since CMake 3.17 list(APPEND CMAKE_CTEST_ARGUMENTS --repeat until-pass:2) if(NOT "$ENV{CIRRUS_CI}") list(APPEND CMAKE_CTEST_ARGUMENTS -j2) endif() endif() endif() if(BUILD_DOCS) set(SPHINX_OPTIONS "" CACHE STRING "additional sphinx-build command line options") set(SPHINX_THEME "sphinx_rtd_theme" CACHE STRING "sphinx HTML theme") set(SPHINX_THEME_OPTIONS "" CACHE STRING "sphinx HTML theme options") set(SPHINX_EXTENSIONS "" CACHE STRING "comma separated list of quoted sphinx extensions") set(SPHINX_CONF_APPEND "" CACHE STRING "append verbatim code to sphinx' conf.py") endif() libmemcached-1.1.4/CMakeLists.txt000066400000000000000000000042721440143131000166460ustar00rootroot00000000000000cmake_minimum_required(VERSION 3.9...3.18) if(${CMAKE_VERSION} VERSION_LESS 3.12) cmake_policy(VERSION ${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}) endif() include(CMakeVersions.txt) project(libmemcached-awesome VERSION "${LIBMEMCACHED_VERSION}" DESCRIPTION "libmemcached-awesome, a C/C++ memcached client library" ) set(PROJECT_HOMEPAGE_URL "https://github.com/awesomized/libmemcached") set(PROJECT_CONTACT "Michael Wallner ") set(CXX_STANDARD 11) set(CMAKE_POSITION_INDEPENDENT_CODE ON) set(GLOBAL_DEFINITIONS _GNU_SOURCE) include(CMakeConfig.txt) include(GNUInstallDirs) list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/CMake") if(BUILD_DOCS OR BUILD_DOCSONLY) add_subdirectory(docs) endif() if(NOT BUILD_DOCSONLY) include(CMake/_Include.cmake) set(CLIENTS capable cat cp dump error exist flush parse ping rm slap stat touch ) add_subdirectory(include) add_subdirectory(src) add_subdirectory(contrib) add_subdirectory(support) # tests need c++17 support add_subdirectory(test) # keep last configure_file(${CONFIGURE_FILE_IN} ${CONFIGURE_FILE_OUT} @ONLY) endif() list(APPEND PROJECT_CONFIG ${CMAKE_SYSTEM_NAME} ${CMAKE_SYSTEM_PROCESSOR}) list(APPEND PROJECT_CONFIG ${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION}) list(APPEND PROJECT_CONFIG ${CMAKE_BUILD_TYPE}) include(CPack.txt) install(FILES AUTHORS BUGS.md ChangeLog-0.md ChangeLog-1.0.md ChangeLog-1.1.md CONTRIBUTING.md LICENSE NEWS README.md TODO COMPONENT doc DESTINATION ${CMAKE_INSTALL_DOCDIR}/ ) if(NOT WIN32) # skip links on windows (cmake bug?) install(FILES ChangeLog ChangeLog.md COPYING COMPONENT doc DESTINATION ${CMAKE_INSTALL_DOCDIR}/ ) endif() if(ENV{INVALID_CONFIGURATION}) message(FATAL_ERROR "invalid configuration -- giving up") endif() libmemcached-1.1.4/CMakeVersions.txt000066400000000000000000000020371440143131000173550ustar00rootroot00000000000000# # libmemcached # set(LIBMEMCACHED_VERSION 1.1.4) set(LIBMEMCACHED_VERSION_INC 1.0) set(LIBMEMCACHED_VERSION_HEX 0x001001004) # libmemcached.so set(LIBMEMCACHED_SO_SOVERSION 11) set(LIBMEMCACHED_SO_VERSION ${LIBMEMCACHED_SO_SOVERSION}.0.0) # # libmemcachedutil # set(LIBMEMCACHEDUTIL_VERSION 1.1.0) set(LIBMEMCACHEDUTIL_VERSION_INC 1.0) set(LIBMEMCACHEDUTIL_VERSION_HEX 0x001001000) # libmemcachedutil.so set(LIBMEMCACHEDUTIL_SO_SOVERSION 2) set(LIBMEMCACHEDUTIL_SO_VERSION ${LIBMEMCACHEDUTIL_SO_SOVERSION}.0.0) # # libmemcachedprotocol # set(LIBMEMCACHEDPROTOCOL_VERSION 0.1.0) set(LIBMEMCACHEDPROTOCOL_VERSION_INC 0.0) set(LIBMEMCACHEDPROTOCOL_VERSION_HEX 0x000001000) # libmemcachedprotocol.so set(LIBMEMCACHEDPROTOCOL_SO_SOVERSION 0) set(LIBMEMCACHEDPROTOCOL_SO_VERSION ${LIBMEMCACHEDPROTOCOL_SO_SOVERSION}.0.0) # # libhashkit # set(LIBHASHKIT_VERSION 1.1.0) set(LIBHASHKIT_VERSION_INC 1.0) set(LIBHASHKIT_VERSION_HEX 0x00100100) # libhashkit.so set(LIBHASHKIT_SO_SOVERSION 2) set(LIBHASHKIT_SO_VERSION ${LIBHASHKIT_SO_SOVERSION}.0.0) libmemcached-1.1.4/CONTRIBUTING.md000066400000000000000000000036161440143131000163400ustar00rootroot00000000000000# Contributor Code of Conduct As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality. Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery * Personal attacks * Trolling or insulting/derogatory comments * Public or private harassment * Publishing other's private information, such as physical or electronic addresses, without explicit permission * Other unethical or unprofessional conduct. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team. This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers. This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.2.0, available at http://contributor-covenant.org/version/1/2/0/. libmemcached-1.1.4/COPYING000077700000000000000000000000001440143131000161342LICENSEustar00rootroot00000000000000libmemcached-1.1.4/CPack.txt000066400000000000000000000062621440143131000156310ustar00rootroot00000000000000# default options set(CPACK_ARCHIVE_COMPONENT_INSTALL ${CPACK_COMPONENT_INSTALL}) set(CPACK_SOURCE_IGNORE_FILES "/[.]git/;/[.](idea|settings|c?project);~$;[.]log$;[.]bak$") macro(cpack_include_if GENERATOR) message(STATUS "Checking ${GENERATOR} package configuration ...") if(EXISTS ${CMAKE_SOURCE_DIR}/CPack/${GENERATOR}.txt) include(${CMAKE_SOURCE_DIR}/CPack/${GENERATOR}.txt) endif() endmacro() # shell installer set(CPACK_BINARY_STGZ ${UNIX}) # binary archive set(CPACK_BINARY_TBZ2 0) set(CPACK_BINARY_TGZ ${UNIX}) set(CPACK_BINARY_TXZ 0) set(CPACK_BINARY_TZ 0) set(CPACK_BINARY_ZIP ${WIN32}) set(CPACK_BINARY_NSIS ${WIN32}) set(CPACK_BINARY_NUGET 0) # source archive set(CPACK_SOURCE_TBZ2 0) set(CPACK_SOURCE_TGZ ${UNIX}) set(CPACK_SOURCE_TXZ 0) set(CPACK_SOURCE_TZ 0) set(CPACK_SOURCE_ZIP ${WIN32}) # project internals set(CPACK_PACKAGE_LICENSE "BSD-3-Clause") set(CPACK_PACKAGE_CONTACT "${PROJECT_CONTACT}") set(CPACK_PACKAGE_VENDOR "${PROJECT_CONTACT}") set(CPACK_PACKAGE_DESCRIPTION "libmemcached-awesome is an open source C/C++ client library and tools for the memcached server (http://memcached.org/). It has been designed to be light on memory usage, thread safe, and provide full access to server side methods.") set(CPACK_PROJECT_CONFIG ${PROJECT_CONFIG}) set(CPACK_PROJECT_CONFIG_FILE "${CMAKE_SOURCE_DIR}/CPack/ProjectConfig.txt") set(CPACK_PACKAGE_DESCRIPTION_FILE "${CMAKE_SOURCE_DIR}/README.md") set(CPACK_PACKAGE_INSTALL_DIRECTORY "${PROJECT_NAME}/${PROJECT_VERSION}") set(CPACK_PACKAGE_CHECKSUM SHA1) set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_SOURCE_DIR}/LICENSE") set(CPACK_RESOURCE_FILE_README "${CMAKE_SOURCE_DIR}/README.md") set(CPACK_CHANGELOG_FILE "${CMAKE_SOURCE_DIR}/ChangeLog-1.1.md") set(CPACK_PACKAGE_DIRECTORY "${CMAKE_BINARY_DIR}") execute_process( COMMAND git describe --tags --match [0-9]*.* WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} OUTPUT_VARIABLE PROJECT_TAG ERROR_VARIABLE GIT_ERROR_OUTPUT ) if(GIT_ERROR_OUTPUT) message(NOTICE "git describe: ${GIT_ERROR_OUTPUT}") endif() string(STRIP "${PROJECT_TAG}" PROJECT_TAG) if(PROJECT_TAG) set(CPACK_PACKAGE_FILE_NAME ${PROJECT_NAME}-${PROJECT_TAG}) set(CPACK_PACKAGE_VERSION ${PROJECT_TAG}) else() set(CPACK_PACKAGE_FILE_NAME ${PROJECT_NAME}-${PROJECT_VERSION}) set(CPACK_PACKAGE_VERSION ${PROJECT_VERSION}) endif() # dependencies if(HAVE_LIBSASL) string(APPEND CPACK_PACKAGE_DEPENDS " libsasl2") endif() if(HAVE_TBB) string(APPEND CPACK_PACKAGE_DEPENDS " tbb") endif() if(HAVE_LIBEVENT) string(APPEND CPACK_PACKAGE_DEPENDS " libevent") endif() # DEBs find_program(DPKG dpkg) if(DPKG) cpack_include_if(DEB) endif() # WIN if(WIN32) cpack_include_if(NSIS) endif() # RPMs find_program(RPMBUILD rpmbuild) if(RPMBUILD) cpack_include_if(RPM) endif() # keep last include(CPack) set(PUSH_ARTIFACTS_SH "${CMAKE_SOURCE_DIR}/scripts/push-artifacts.sh") if(WIN32) set(PUSH_ARTIFACTS_CMD msys2 -c '${PUSH_ARTIFACTS_SH} ${CPACK_PACKAGE_VERSION}') else() set(PUSH_ARTIFACTS_CMD ${PUSH_ARTIFACTS_SH} ${CPACK_PACKAGE_VERSION}) endif() add_custom_target(push-artifacts COMMAND ${PUSH_ARTIFACTS_CMD} WORKING_DIRECTORY ${CPACK_PACKAGE_DIRECTORY} ) libmemcached-1.1.4/CPack/000077500000000000000000000000001440143131000150625ustar00rootroot00000000000000libmemcached-1.1.4/CPack/DEB.txt000066400000000000000000000004121440143131000162120ustar00rootroot00000000000000set(CPACK_BINARY_DEB ON) set(CPACK_DEB_COMPONENT_INSTALL ${CPACK_COMPONENT_INSTALL}) set(CPACK_DEBIAN_PACKAGE_DEPENDS "${CPACK_PACKAGE_DEPENDS}") set(CPACK_DEBIAN_ENABLE_COMPONENT_DEPENDS ON) set(CPACK_COMPONENT_bin_DEPENDS lib) set(CPACK_COMPONENT_dev_DEPENDS lib) libmemcached-1.1.4/CPack/NSIS.txt000066400000000000000000000004111440143131000163730ustar00rootroot00000000000000set(CPACK_PACKAGE_INSTALL_DIRECTORY "${CPACK_PACKAGE_FILE_NAME}") set(CPACK_NSIS_MODIFY_PATH ON) set(CPACK_NSIS_DISPLAY_NAME "${CPACK_PACKAGE_FILE_NAME}") set(CPACK_NSIS_PACKAGE_NAME "${CPACK_PACKAGE_FILE_NAME}") set(CPACK_NSIS_HELP_LINK "${PROJECT_HOMEPAGE_URL}") libmemcached-1.1.4/CPack/ProjectConfig.txt000066400000000000000000000002211440143131000203520ustar00rootroot00000000000000string(REPLACE ";" "-" CPACK_PACKAGE_DIRECTORY "${CPACK_PACKAGE_DIRECTORY}/${CPACK_PACKAGE_VERSION}/${CPACK_GENERATOR}/${CPACK_PROJECT_CONFIG}") libmemcached-1.1.4/CPack/RPM.txt000066400000000000000000000005711440143131000162640ustar00rootroot00000000000000set(CPACK_BINARY_RPM ON) set(CPACK_RPM_COMPONENT_INSTALL ${CPACK_COMPONENT_INSTALL}) set(CPACK_RPM_PACKAGE_DEPENDS "${CPACK_PACKAGE_DEPENDS}") set(CPACK_RPM_PACKAGE_LICENSE "${CPACK_PACKAGE_LICENSE}") #set(CPACK_RPM_CHANGELOG_FILE "${CPACK_CHANGELOG_FILE}") set(CPACK_RPM_PACKAGE_DESCRIPTION "${CPACK_PACKAGE_DESCRIPTION}") set(CPACK_RPM_PACKAGE_GROUP "Development/Libraries") libmemcached-1.1.4/ChangeLog000077700000000000000000000000001440143131000200172ChangeLog.mdustar00rootroot00000000000000libmemcached-1.1.4/ChangeLog-0.md000066400000000000000000000436641440143131000164240ustar00rootroot00000000000000# ChangeLog v0.x ## v 0.53 > released 2011-09-27 * Fix for FreeBSD/OpenBSD and -lm * Added memcached_exist() * Fix for memory when using config test. * CLI gained --quiet ## v 0.52 > released 2011-09-12 * Build fixes for Ubuntu/Suse. * Fixes for OSX Lion. * Bug fix for looping back through dns lookups under certain failures. * Fixes related to dead server failures. ## v 0.51 > released 2011-07-21 * memcached_callback_set() now takes its data argument as const * Update to tests. * Fix in parser for port number. ## v 0.50 > released 2011-06-20 * Updates to C++ interface * Custom free allocators need to now check for value before calling free. * memcached_fetch_result() now uses the internal result when available (about 25 to 50% faster). * Fix for stats structure. * Updates to documentation. * memcached_fetch_result() now uses the internal result when available (about 25 to 50% faster). ## v 0.49 > released 2011-04-14 * Fix calls to auto methods so that if value is not passed in nothing bad happens. * New parser calls for generating memcached_st objects. * New error system. * New flow control for messages means faster get/set calls. * Added new documentation system. * A behavior change has been now made that if you specify a weight for any server, we enable the weight flag and do weight balancing. * Added MEMCACHED_BEHAVIOR_REMOVE_FAILED_SERVERS to simplify the setting of AUTO REJECT for servers. ## v 0.48 > released 2011-03-16 * Fix memory leak in server parse. * Move test framework out to be its own library (easier to work with Gearman). ## v 0.47 > released 2011-02-24 * Additional fixes for OpenBSD. * Bug fix 677609, 456080. * SIGPIPE fix for Linux send(). * memcapable can now test ascii or binary based on flags. * Additional build fixes for SASL. ## v 0.46 > released 2011-02-14 * Fixes a number of corner case bugs. * Fixes related to OpenBSD. * Better testing for protocol version. * Removes special case infinite wait on blocking setup. ## v 0.45 > released 2011-02-09 * Add support for systemtap ## v 0.44 > released 2010-09-23 * Windows bug fixes. * Hudson port support in test harness. * Improved portability of test hanrness. * SASL fixes. ## v 0.43 > released 2010-07-28 * Added --args to memstat so that a greater range of values can be returned. * Prelimanary support for Windows. * memcached_stat_execute() merged. ## v 0.42 > released 2010-07-06 * Mistake in libtool caused issue with library version ## v 0.41 > released 2010-06-30 * Added --file for memcat. * Added limemcached_ping() to libmemcached_util * Bugfix for some cases where connect would have issues with timeout. * Wrong value for errno given as error on an IO failure inside of poll. * Bug fix for issue where multiple interfaces with bad DNS were not being caught. ## v 0.40 > released 2010-04-23 * Placed retry logic in for busted resolvers * Add an ignore for SIGPIPE to solve OSX issues. * A couple of fixed for memcached_light server. * Updated to debug mode to track io_wait ## v 0.39 > released 2010-04-06 * Add support for prefix keys to binary protocol. * Remove the undocumented call memcached_server_remove(). * The undocumented call memcached_server_by_key() now returns const. * memcached_server_error_reset() has been deprecated. * memcached_server_list() has been deprecated. Use memcached_server_cursor() to walk the servers found in a memcached_st() structure. * memcached_verbosity() can now be run concurrently with other operations. * SASL support. * Fixes memory leak found in EJECT HOSTS. ## v 0.38 > released 2010-02-10 * C++ interface for libhashkit. * Modified memcached_set_memory_allocators() so that it requires a context pointer. * memcached_clone() now runs 5 times faster. * Functions used for callbacks are now given const memcached_st. * Added MEMCACHED_BEHAVIOR_CORK. * memslap now creates a configuration file at ~/.memslap.cnf * memcached_purge() now calls any callbacks registered during get execution. * Many fixes to memslap. * Updates for memcapable. * Compile fixes for OpenBSD. * Fix for possible recursive decent on IO failure. ## v 0.37 > released 2010-01-12 * Fixed build for libhashkit. * Fixed install path regression. * Modified RPM to strict check install. * Added documentation for memcached_server_cursor(); * Added memcached_servers_reset(). * Modified memcached_st to remove dead cursor_server member. ## v 0.36 > released 2010-01-07 * Merged in new memslap utility. * All of constants.h has been updated to match style (all old identifiers continue to work). * Added first pass for libhashkit. * Updated test Framework/extended tests. * Random read support during replication added. * Modified use_sort so that the option can be applied to any distribution type. * We removed the MEMCACHED_BEHAVIOR_KETAMA_COMPAT_MODE added in 0.35. Instead use memcached_behavior_set_distribution(). ## v 0.35 > released 2009-11-09 * Added support for by_key operations for inc/dec methods. * Added mget test to memslap. * Support for compatible ketama for SpyMemcached * Update C++ interface. * Fix for memcp ## v 0.34 > released 2009-10-13 * Added support for setting behavior flags on a connection pool. * Don't increment server_failure_counter on normal disconnects. * Added prototype for a callback based protocol parser (server side) with examples so that you could let your own application speak the memcached protocol * Updated memcapable to test ASCII protocol. * Changed behavior so that server can be removed at first sign of failure. * Added memcached_server_get_last_disconnect() call ## v 0.33 > released 2009-09-23 * Added memcapable to test servers for binary compatibility. * Updated C++ interface. Added basic support for C++ exceptions. Added multiple constructors the memcached client object. The C++ interface now takes parameters which are C++ types (such as std::string). * Several bug fixes for binary protocol support. * Fixed crashing issue with dumping from memcachd server (server internals were changed without documenting change). ## v 0.32 > released 2009-09-15 * Change of behavior where linger is only modified for no-block and then it is set to zero. * Added Twitter's memcached_server_error() functions. * Fix for OSX compiles in development builds. * Updated C++ interface. * Updated memcached_mget and memcached_mget_by_key to take a size_t as a parameter instead of an unsigned int for number_of_keys. ## v 0.31 > released 2009-07-10 * Added support or HA via replication. * malloc() removed for server key usage. * Update build system. * Added support for memcached_set_memory_allocators(). * Fixed bug in configure.ac for have_htoll. ## v 0.30 > released 2009-06-01 * Added memcachd_dump command (and framework for memdump tool). * Realigned all structures to remove padding (and line up important bits for 64bit caches. * Remove some of sprintf() in storage calls(). * Removed printf() in stat call for unknown stat member. * memcached_generate_hash() function added. * Added tests to make sure all hash functions are stable. ## v 0.29 > released 2009-05-19 * Fixed malloc usage to calloc for spots where we need zero filled memory. * All code warnings now treated as errors. * Fixes for debian packaging. * Added new pooling mechanism. * MEMCACHED_BEHAVIOR_NO_BLOCK no longer also sets MEMCACHED_BEHAVIOR_BUFFER_REQUESTS. * Updated generic rpm. ## v 0.28 > released 2009-04-15 * Fixed bug in init sructure (reapplied) * Fixed bug in get/set by key (nikkhils@gmail.com) ## v 0.27 > released 2009-03-30 * Added new UDP fire-forget mode. * Reworked performance for mget() to better make use of async protocol * Cleaned up execution of fetch (just one set of code now) * Fixed Jenkin's for big endian hosts. * Updates for memstat to determine network latency. * Updates for binary protocol. * Many updates to documentation. ## v 0.26 > released 2009-01-29 * Fix for decrement on hash key * Fixed assert that was catching bad memset() call in host_reset() * Fix purge issue for blocked IO which has been stacked. ## v 0.25 > released 2008-11-28 * Jenkins HASH added. * Update of Murmur hash code * Support explicit weights (Robey Pointer, Evan Weaver) * Bugfix for ketama continuum (Robey Pointer) * New behavior MEMCACHED_BEHAVIOR_HASH_WITH_PREFIX_KEY (Robey Pointer) * Don't ever call stats for weighting servers, because it is unstable. ## v 0.24 > released 2008-09-16 * Cleanup compile warnings. * Fix issues in partitioning by keys. * Fixed "fail case" to make sure when calling memcached_clone() no memcached_st is over written. * New memcached_server_by_key() method for finding a server from a key. * memcached_server_free() was added for freeing server structures. ## v 0.23 > released 2008-09-07 * Added strings.h header for Solaris 9 * Solaris 64bit fix. * Support for weighted Ketama from Yin Chen. * Fix for Chinese * Fix for 0 length key to trigger bad key. * Added behaviors MEMCACHED_BEHAVIOR_SND_TIMEOUT, MEMCACHED_BEHAVIOR_RCV_TIMEOUT * Support for Binary Protocol added ## v 0.22 > released 2008-07-14 * Fix where master key was no being checked for "bad key" * Fixed bugs in stats output (thread output was wrong) * Clarified MEMCACHED_BAD_KEY_PROVIDED is return for bad prefix key. * Found a bug in Flags return (Jacek Ostrowski) * Fixed issue with compiling on Visual Studio ## v 0.21 > released 2008-05-24 * Change of char * to const char * for all key based functions. * New MEMCACHED_CALLBACK_PREFIX_KEY added. You can now create domains for values. * Fixed bug introducd in last version on memcp * Fix for death of file io to call shutdown() ## v 0.20 > released 2008-05-05 * New consistent distribution tests. * Found a memory leak when a server constantly fails. * Fix in watchpoint macro * Changed default timeout to 1 second for poll timeouts * Wheel uses less memory/dynamic allocation for size (no longer limited to 512 hosts by default. * memslap memory leak fix * Added Ketama distribution * Fix assert.h compile problem on CentOS ## v 0.19 > released 2008-04-09 * Documentation fix in libmemcached. * Fixed bug where sort was always occuring on hosts * Logic fix in branch prediction (thanks Jay!) * Read through cached support. * Fixed for cas by key operation. * Fix for memcached_server_st list structures to have correct count. * Added callback MEMCACHED_CALLBACK_DELETE_TRIGGER * Removed function call in favor of macro (aka cut out some instructions) ## v 0.18 > released 2008-03-17 * Fix plus tests for non-zero value objects and flags. * MEMCACHED_HASH_MURMUR added for murmur algorithm provided. * MEMCACHED_BEHAVIOR_RETRY_TIMEOUT added to keep connecting from looping on timeout. * gcc branch prediction optimizations * Refactored entire tree to make include files cleaner * Fixed leaked socket. ## v 0.17 > released 2008-02-27 * MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT added for connect timeout in non-block mode. * Incompatible change in memcached_behavior_set() api. We now use a uint64_t, instead of a pointer. * Fix for storage of values for zero. * memcached_server_cursor() function added to API for cycling through servers. ## v 0.16 > released 2008-02-18 * Work on the UDP protocol * Added get_by_key, set_by_key tests for C++ API * Fix for limit_maxbytes to be 64bit in stats * Added Atom Smasher test (scale baby, scale!) * Servers are now sorted, meaning that servers are now ordered so that clients with the same lists, will have same distribution. (Idea from Ross McFarland). MEMCACHED_BEHAVIOR_SORT_HOSTS was added to enable this support. * Added MEMCACHED_BAD_KEY_PROVIDED error for auto, set, and get operations. MEMCACHED_BEHAVIOR_VERIFY_KEY was added to enable this feature. * More error messages on command line tools. * Fixed bugs in memcached_cas() operator. * Fix to loop through interfaces ## v 0.15 > released 2008-01-29 * More work on the C++ API. * Bug fixes around block corner cases. * Slight performance increase in both read() and write(). ## v 0.14 > released 2008-01-22 * For for bug found by Evan Weaver where increment() was not returning propper error of value was not found. * Fix for bad null pointer on flag by Toru Maesaka. * Refactor of all IO to just pass in the active server * Problem configuring (PKG_CHECK_MODULES) fixed by removal of "rpath" in support/libmemcached.pc.in (Thanks to Ross McFarland). * Added memcached_callback_get()/set() * First prototype of C++ interface * Updated docs for uint16_t changes in previous release ## v 0.13 > released 2008-01-13 * MEMCACHED_BEHAVIOR_USER_DATA added to store user pointer. * Fix for failure to connect to invalidate socket. * Patch from Marc Rossi to add --hash option for memcp, memrm, and memcat. * Kevin's patch for fixing EOF issues during a read. * Toru Maesaka patch for stats mismatch * Fix for when CRC return 0 * Fixed uint16_t issues around flags. Turns out the documentation on the protocol was wrong. * Lingering socket fixes for FreeBSD. * Patches from Kevin Dalley for FreeBSD 4.0 * Added multi delete functions. * All get key returns have C style null termination * If memcached_server_list_append is passed NULLs instead of pointers it returns NULL. * Added memcached_fetch_execute() method * Found a bug where memcached_fetch() was not null terminating the result value. * memcached_behavior() now has the ability to set "buffering" so that data is not automatically flushed. * Behavior change, buffered commands now return MEMCACHED_BUFFERED ## v 0.12 > released 2007-12-11 * Updates for consistent hashing * IPV6 support * Static allocation for hostname (performance) * Fixed bug where in non-block mode all data might not have been sent on close(). * Refactor of memcached_get() to use common code. * Change in value fetch, MEMCACHED_END is now returned when keys are no longer in the pipe. * Fixed bug where key could be out of range of characters * Added _by_key() methods to allow partitioning of values to particular servers. * MEMCACHED_DEFAILT_TIMEOUT is now set to a non -1 value. * Performance improvements in get operations. ## v 0.11 > released 2007-11-26 * Added option to memcache_behavior_set() so that poll() can be timed out. * Fixed memory leak in case of using memcached_fetch_result() where no value was returned. * Bug fixed in memcached_connect() which would cause servers that did not need to be enabled to be enabled (performance issue). * Rewrote bounds checking code for get calls. * "make test" now starts its own memcached servers. * Added Hseih hash (MEMCACHED_HASH_HSIEH), which is showing about 7% performance over standard hash. ## v 0.10 > released 2007-11-21 * Added append binary test. * Added MEMCACHED_BEHAVIOR_CACHE_LOOKUPS behavior so that you can save on multiple DNS lookups. * Added CAS support, though this is optional and must be enabled during runtime. * Added the utility memerror to create human readable error strings from memcached errors (aka convert ints to strings) * Fixed type in MEMCACHED_HOST_LOOKUP_FAILURE * Fixed bug where hostname might not be null terminated * Moved to using gethostbyname_r() on Linux to solve thread safety issue * Added -rpath support for pkg-config * Documentation fix for hash setting using memcached_behavior_set() ## v 0.9 > released 2007-11-15 * fix for when no servers are definied. * different buffers are now kept for different connections to speed up async efforts * Modified increment/decrement functions to return uint64_t values * Fixed bug in cases where zero length keys were provided * Thread cleanup issue in memslap * No hostname lookup on reconnect * Fix for flag settings (was doing hex by accident!) * Support for 1.2.4 server additions "prepend" and "append" added. * Added memcached_version()... not sure if I will make this public or not. ## v 0.8 > released 2007-11-05 * Adding support for CRC hash method * Adding support for UNIX sockets * Added additional HASHing methods of FNV1_64,FNV1A_64, FNV1_32, FNV1A_32 * Added pkgconfig support (PKG_CHECK_MODULES) * Fixed conflict with defined type in MySQL * Added memcached_result_st structure and functions to manipulate it. ## v 0.7 > released 2007-10-30 * Poved to poll() from select() * Fixes in internal string class for allocation of large numbers of strings. * memcached_mget() function now sends keys as it parses them instead of building strings as it goes. * Propper flush now for making sure we get all IO sent even when in non-block mode. * Added --enable-debug rule for configure * All asserts() removed (hey this is going into production!) ## v 0.6 > released 2007-10-17 * get value returns are now null terminated (request by Cal Heldenbrand) * Fixed connections for more hosts then two. * Rewrite of the read/write IO systems to handle different sorts of host failures. * Added man pages for all functions and tools * Raised buffer size for readinng/writing to 16K * You can now optionally set the socket size for recv/send via memached_behavior_set/get. ## v 0.5 > released 2007-10-09 * Ruby maintainer mentioned TCP_NODELAY patch he had added. Added this to C library as well. (Eric Hodel drbrain@segment7.net) * Added support script for set_benchmark * Updated memslap to allow testing of TCP_NODELAY * Updated memslap to support --flush (aka dump memcache servers before testing) * Fixed bug in multiple hosts not being activated * Added environmental variable MEMCACHED_SERVERS which can be used to set the servers list. * fixed memcached_stat method (and now memstat works) * server connect now happens on demand. * Help for all command line applications ## v 0.4 > released 2007-10-03 * Added buffered IO to write calls for keys * Added buffered IO for reads * memstat was broken (bad if/else on connect) * New non-blocking IO (not default yet). Mucho faster * Refactor of test system. * memslap crash solved ## v 0.3 > released 2007-10-01 * Jeff Fisher provided a spec file * Added "make rpm" around dist file * Added support for Solaris * Added support for DTrace * Fixed read to be recv and write to be send * Bug fix where memstat would core if no server was found * Added memslap tool (load generator) * Numerous bug fixes in library * Added calls to library for creating host lists (see text cases to understand how to use this). ## v 0.2 > released 2007-09-27 * First public version libmemcached-1.1.4/ChangeLog-1.0.md000066400000000000000000000057151440143131000165560ustar00rootroot00000000000000# ChangeLog v1.0 ## v 1.0.18 > released 2014-02-09 * MEMCACHED_BEHAVIOR_RETRY_TIMEOUT can now be set to zero. * Numerous bug fixes. ## v 1.0.17 > released 2013-04-03 * Remove c++ namespace that was being exposed (the API should be plug compatible).. * Fix cases where --servers wasn't behaving the same in all clients. ## v 1.0.16 > released 2013-02-01 * Added support to do two part shutdown of socket. * Fixes for Fedora 18. * Fix for binary memcached_touch() ## v 1.0.15 > released 2012-12-17 * Added support for Murmur3 (HASHKIT_HASH_MURMUR3) * Portability fixes. ## v 1.0.14 > released 2012-11-14 * CLIENT_ERROR fixed to not be treated as a fatal error. * Compiler fixes for older Ubuntu releases. ## v 1.0.13 > released 2012-10-19 * Fix bug that caused version string to not be exported correctly. ## v 1.0.12 > released 2012-10-09 * Added memcached_result_take_value(). * Added ax_libmemcached.m4 ## v 1.0.11 > released 2012-09-17 * Removed custom version of memcached. * Updated hardening rules. * Fixed a case where the return error from a socket connection differred from that of a TCP/IP socket. ## v 1.0.10 > released 2012-07-30 * --disable-assert has been removed from configure, and --enable-assert has been added in its place. * Compiling fixes for Clang on OSX Mountain Lion. ## v 1.0.9 > released 2012-07-05 * Faster close on socket. * Instance allocation is now seperated from server interface. This should allow for a better preservation of ABI compliance from now on. * Fix close on exec bug. * Numerous other bug fixes. ## v 1.0.8 > released 2012-05-22 * Added support for setting options via ENV variable LIBMEMCACHED * Fix corner case on last used result. ## v 1.0.7 > released 2012-04-28 * Add API call for exist calls. * Update all license files to be BSD. ## v 1.0.6 > released 2012-04-08 * Fixes for gcc 4.7, lp:961812 * Fix for restart issue that happens under testing. * Fix for lp:962815. * Support for transparent AES encryption. ## v 1.0.5 > released 2012-03-14 * Fixes for OSX. * Version is now parsed directly in the parser, which makes buffered operations now work with it.. * memstat has been extended so that it can be used to find the version of the server. * Update documentation. * Fixes for compile issues on Debian and Ubuntu ## v 1.0.4 > released 2012-01-27 * Fix for memcached_dump(). * Additional testing for memcached_stat_execute(). ## v 1.0.3 > released 2012-01-09 * Increased size of sort buffer used during Ketama. * Added support for new behavior to handle dead servers. * Overall haul of UDP IO. * Fixed C compile issue with memcached_exist() * Numerous bug fixes. * Clang support for OSX. * All commands now using vector send support. ## v 1.0.2 > released 2011-10-24 * Dropped libmemcached/memcached_util.h (undocumented header file) * Added memcached_touch() and memcached_touch_by_key() * UDP support restructured to toggle on a complete memcached_st structure. --- See [ChangeLog-0](./ChangeLog-0.md) for changes prior v1.0. libmemcached-1.1.4/ChangeLog-1.1.md000066400000000000000000000231241440143131000165510ustar00rootroot00000000000000# ChangeLog v1.1 ## v 1.1.4 > released 2022-03-06 * Fix [gh #107](https://github.com/awesomized/libmemcached/issues/107): macOS: deprecated sasl API (improve detection of `libsasl2`). * Fix [gh #131](https://github.com/awesomized/libmemcached/issues/131): Consider renaming tools (add `CLIENT_PREFIX` build option; default: `mem`) * Fix [gh #132](https://github.com/awesomized/libmemcached/issues/132): Add build of static library (add `BUILD_SHARED_LIBS` build option; default: `ON`). * Fix [gh #134](https://github.com/awesomized/libmemcached/issues/134): Update client option documentation. * Fix [gh #136](https://github.com/awesomized/libmemcached/issues/136): `libmemcachedutil` is underlinked (link against libmemcached). * Fix [gh php-memcached#531](https://github.com/php-memcached-dev/php-memcached/issues/531): `get` returns random values when lower than default `OPT_POLL_TIMEOUT` is set. ## v 1.1.3 > released 2022-11-09 * Fix [gh #130](https://github.com/awesomized/libmemcached/issues/130) with [gh #124](https://github.com/awesomized/libmemcached/issues/124): Server response count can underflow. ## v 1.1.2 > released 2022-08-10 * Fix handling of negative expiration values, which are somehow allowed by legacy. See also [gh #125](https://github.com/awesomized/libmemcached/issues/125), and [gh #76](https://github.com/awesomized/libmemcached/issues/76). * Fix [gh #122](https://github.com/awesomized/libmemcached/issues/122): If libcrypto implementation of AES is used, do not compile internal. * Fix missing include of in tests. * Fix warnings with non-SASL builds. * Fix pthread.h detection. ## v 1.1.1 > released 2021-09-16 * Fix [gh #67](https://github.com/awesomized/libmemcached/issues/67): GET returns `NOTFOUND` on `TIMEOUT`. * Fix [gh #113](https://github.com/awesomized/libmemcached/issues/105): Build failure with Catch2 < 2.13.5. * Add [gh #114](https://github.com/awesomized/libmemcached/pull/114): Add possibility to use libcrypto for encryption. * Add [gh #115](https://github.com/awesomized/libmemcached/pull/115): Add `LIBMEMCACHED_AWESOME` CPP define. * Add test for [gh #75](https://github.com/awesomized/libmemcached/issues/75): memcached_clone of SASL connection closes random file descriptor. * Fix [gh #116](https://github.com/awesomized/libmemcached/issues/116): Add libmemcachedpotocol-0-0/configure.h guarding `ssize_t` typedef. * Fix [gh #120](https://github.com/awesomized/libmemcached/issues/120): libmemcached.pc is missing a `Requires` entry for libsasl2. ## v 1.1.0 > released 2021-06-23 **Changes from beta3:** * Add ASCII multi get support to bin/memslap. See logs from `beta3`, `beta2`, and `beta1` for the full list of changes since the last 1.0 release. ## v 1.1.0-beta3 > released 2021-04-15 **Changes from beta2:** * Fix [gh #108](https://github.com/awesomized/libmemcached/issues/105): macOS Big Sur: dtrace does not understand -G switch. * Add support for IPv6 bracketed syntax in `memcached_servers_parse`. * Make `memcat`'s `--file` option's argument optional defaulting to ``. * Fix libmemcachedprotocol's binary `STAT` and `VERSION` handlers. * Fix [gh #105](https://github.com/awesomized/libmemcached/issues/105): EINTR handled too defensively when polling. ## v 1.1.0-beta2 > released 2020-12-28 **Changes from beta1:** * Fix [gh #103](https://github.com/awesomized/libmemcached/issues/103): Build failure on 32-bit. * Fix [gh #102](https://github.com/awesomized/libmemcached/issues/102): Doc build with old sphinx. * Fix [gh #100](https://github.com/awesomized/libmemcached/issues/100): Revert symbolic rename of public header include directories. * Fix [gh #98](https://github.com/awesomized/libmemcached/issues/98): Library SONAMEs and NAME_LINKs differ from 1.0.18. * Fix [gh #97](https://github.com/awesomized/libmemcached/issues/97): Location of cmake files installation directory. * Fix [gh #96](https://github.com/awesomized/libmemcached/issues/96): LIBXXX_VERSION_HEX constants format. ## v 1.1.0-beta1 > released 2020-12-21 **NOTE:** This is a bug fix release, not a feature release. The minor version number was incremented due to the following changes: * Ported build system to CMake. * Ported test suite to Catch2. * Build requires C++11 compiler support. * Tests require C++17 compiler support. * Moved to the Semantic Versioning Specification: https://semver.org * Moved the project from launchpad to github: * Source: https://github.com/awesomized/libmemcached * Documentation: https://awesomized.github.io/libmemcached * Continuous Integration: * Github: https://github.com/awesomized/libmemcached/actions (Linux, MacOS, Windows **·** amd64) * Sourcehut: https://builds.sr.ht/~m6w6/libmemcached (FreeBSD, OpenBSD **·** amd64) * Build artifacts: https://artifacts.m6w6.name/libmemcached/ rsync://m6w6.name::artifacts/libmemcached/ * Fix build failure due to comparison of incompatible types in bin/memflush and bin/memstat. * Fix wrong type of memcached_instance_st::server_timeout_counter_query_id from uint32_t to uint64_t. * Fix memcached_dump(): returned MEMCACHED_CLIENT_ERROR on request to dump illegal slab id. * Fix bin/memcapable: failed with "No hostname was provided" when providing a hostname. * Fix hashkit/murmur and hashkit/murur3: undefined behavior on platforms requiring aligned access. * Fix Memcache::set(): possible subscription of empty vector. * Fix libmemcached_util_version_check(). * Fix ketama/consistent hashing: crash on reallocation of continuum. * Fix [gh #90](https://github.com/awesomized/libmemcached/issues/90): Build fails on Darwin. * Fix [gh #83](https://github.com/awesomized/libmemcached/issues/83): memcp waits forever if file no found. * Fix [gh #80](https://github.com/awesomized/libmemcached/issues/80): memparse docs. * Fix [gh #72](https://github.com/awesomized/libmemcached/issues/72) and [gh #47](https://github.com/awesomized/libmemcached/issues/47): memcached_return_t docs. * Fix [gh #62](https://github.com/awesomized/libmemcached/issues/62): uint32_t overflow cause busy loop. * Removed restriction of UDP+IPv6. * Fix SERVER_ERROR_MEMORY_ALLOCATION_FAILURE: recognize more strings returned by the server. * Fix [gh #13](https://github.com/awesomized/libmemcached/issues/13): reset continuum counter after freeing them. * Fix [gh #14](https://github.com/awesomized/libmemcached/issues/14) and [gh #17](https://github.com/awesomized/libmemcached/issues/17): SASL: AUTH_CONTINUE was considered a failure and caused IO reset. * Fix [gh #25](https://github.com/awesomized/libmemcached/issues/25): hashkit/murmur3 unavailable. * Fix missing handling of EAGAIN for non-blocking unix domain socket. * Fix [gh #35](https://github.com/awesomized/libmemcached/issues/35): handling of BEHAVIOR_REMOVE_FAILED_SERVERS. * Fix [gh #41](https://github.com/awesomized/libmemcached/issues/41): ensure stable sort on continuum host key collision. * Fix [gh #42](https://github.com/awesomized/libmemcached/issues/42): MEMCACHED_MAX_BUFFER docs. * Fix [gh #43](https://github.com/awesomized/libmemcached/issues/43): libmemcached_configuration docs. * Fix [gh #46](https://github.com/awesomized/libmemcached/issues/46): clarification on millisecond timeout docs. * Fix [gh #50](https://github.com/awesomized/libmemcached/issues/50): memcached_fetch_result() can return previously returned data. * Fix [gh #53](https://github.com/awesomized/libmemcached/issues/53): stack overflow in memcached_fetch_result(). * Fix [gh #57](https://github.com/awesomized/libmemcached/issues/57): include vs * Fix [gh #58](https://github.com/awesomized/libmemcached/issues/58): more specific error messages when connect() fails. * Fix [gh #59](https://github.com/awesomized/libmemcached/issues/59): bin/memcat: typo in "No servers provied". * Fix [gh #77](https://github.com/awesomized/libmemcached/issues/77): undeclared UINT64_C in ketama.cc. * Fix [gh #12](https://github.com/awesomized/libmemcached/issues/12): never reconnects after connection reset (binary protocol). * Fix [gh #49](https://github.com/awesomized/libmemcached/issues/49): assertion memcached_failed(rc) failed in memcached_send_ascii(). * Fix [gh #67](https://github.com/awesomized/libmemcached/issues/67): get returns NOTFOUND on timeout. * Fix [gh #76](https://github.com/awesomized/libmemcached/issues/76): memcached_touch() crashes when expiration=-1 (ASCII only). * Fix [gh #23](https://github.com/awesomized/libmemcached/issues/23): build fails with bison 2.3. * Fix memaslap: build fails with newer compiler versions. * Fix usage of strerror_r() implementations returning pointer to char. * Fix pipelining commands with memcached >= 1.6. * Fix memcached_stat_get_value(): buffer overflow. * Fix memcached_stat(): undefined behavior due to unintialized memcached_return_t. * Fix SASL tests: requires SASL_PWDB_CONF. * Fix bin/memaslap to idnentify itself as memaslap instead of memslap. * Fix bin/memcapable to work with memcached >= 1.6. * Fix murmur and murmur3 hashes on big endian platforms. * Fix [gh #82](https://github.com/awesomized/libmemcached/issues/82), [gh #64](https://github.com/awesomized/libmemcached/issues/64) and [gh #21](https://github.com/awesomized/libmemcached/issues/21): clarify documentation on replication. * Fix [gh #95](https://github.com/awesomized/libmemcached/issues/95): MEMCACHED_CALLBACK_GET_FAILURE and MEMCACHED_BEHAVIOR_BUFFER_REQUESTS * Fix bin/memcat to output flags if requested with `--flag`. * Fix [gh #68](https://github.com/awesomized/libmemcached/issues/68): Windows support. --- See [ChangeLog-1.0](./ChangeLog-1.0.md) for changes prior v1.1. libmemcached-1.1.4/ChangeLog.md000077700000000000000000000000001440143131000207132ChangeLog-1.1.mdustar00rootroot00000000000000libmemcached-1.1.4/LICENSE000066400000000000000000000031301440143131000151030ustar00rootroot00000000000000Copyright (c) 2006-2014 Brian Aker, DataDifferential, https://datadifferential.com/ Copyright (c) 2020-2021 Michael Wallner, Awesome Inc, https://awesome.co/ All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. libmemcached-1.1.4/LICENSE.HEADER000066400000000000000000000016121440143131000160350ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020-2021 Michael Wallner https://awesome.co/ | +--------------------------------------------------------------------+ */ libmemcached-1.1.4/NEWS000066400000000000000000000000161440143131000145750ustar00rootroot00000000000000See Changelog libmemcached-1.1.4/README.md000066400000000000000000000104361440143131000153640ustar00rootroot00000000000000# libmemcached-awesome [![License Badge]](https://opensource.org/licenses/BSD-3-Clause) [License Badge]: https://img.shields.io/badge/License-BSD%203--Clause-blue.svg "BSD 3-Clause" libmemcached-awesome is an open source C/C++ client library and tools for the memcached server (http://memcached.org/). It has been designed to be light on memory usage, thread safe, and provide full access to server side methods. > **NOTE:** > This is a resurrection of the original work from Brian Aker at > [libmemcached.org](https://libmemcached.org). ## Documentation [![Docs Actions Badge]]( https://github.com/awesomized/libmemcached/actions?query=workflow%3Adocs-publish-pages) [Docs Actions Badge]: https://github.com/awesomized/libmemcached/workflows/docs-publish-pages/badge.svg?branch=v1.x "Github Docs Action" See https://awesomized.github.io/libmemcached ### Building and updating docs See [gh-pages/publish](./docs/gh-pages/publish.sh) script and the [docs-publish-pages](./.github/workflows/docs-publish-pages.yml) workflow, which automate pushing updated documentation to github pages. ## Installing libmemcached-awesome uses `CMake`. Please see/edit [`CMakeConfig.txt`](./CMakeConfig.txt) or use `ccmake(1)` or `cmake-gui(1)` to set any preferred options. ### From source git clone github.com:awesomized/libmemcached mkdir build-libmemcached cd $_ cmake ../libmemcached make sudo make install #### Requirements * CMake 3.9+ * C++11 compiler * GNU Bison 2.3+ and Flex ##### Optional dependencies * C++17 compiler (required for: tests) * Intel's libtbb (optional for: tests; for GCC's stdlib parallelism support) * pthreads (required for: tests, contrib/bin/memaslap, libmemcachedutil/pool) * libevent (required for: contrib/bin/memaslap) * Cyrus' libsasl2 (required for: libmemcached/sasl) ### Binaries CI and release builds for Linux, a couple BSDs, MacOS and Windows are available at https://artifacts.m6w6.name/libmemcached/ and rsync://m6w6.name::artifacts/libmemcached/. ## Testing Enable the `BUILD_TESTING` setting for a build and run `make test`. cmake -DBUILD_TESTING=ON ../libmemcached make test ### Continuous integration [![Actions Badge]](https://github.com/awesomized/libmemcached/actions?query=workflow%3Acmake-build-ci) [![Sourcehut Badge]](https://builds.sr.ht/~m6w6/libmemcached) [Actions Badge]: https://github.com/awesomized/libmemcached/workflows/cmake-build-ci/badge.svg?branch=v1.x "Github Actions" [Sourcehut Badge]: https://builds.sr.ht/~m6w6/libmemcached/commits.svg "Sourcehut Builds" CI/Testing is performed on the following system matrix: | OS | Compiler | Arch | Comments | |------------------|------------------------------|-------------------------|----------------------------| | Linux | GNU 7/9/10, Clang 9/10/11/12 | amd64 | sasl, coverage, sanitizers | | MacOS | AppleClang 12 | amd64 | sasl, coverage | | FreeBSD | Clang 11 | amd64 | sasl, coverage | | OpenBSD | Clang 10 | amd64 | sasl, coverage | | Windows | MSVC 16, MinGW | amd64 | no sasl, no tests | | Solaris | SunPro 12.5 | amd64 | no sasl, no tests, manually| libmemcached-awesome has been tested against [memcached](https://github. com/memcached/memcached) v1.5 and v1.6. ## ChangeLog Check out the latest [releases](https://github.com/awesomized/libmemcached/releases) or the bundled [ChangeLog](./ChangeLog-1.1.md) for a comprehensive list of changes. ## License libmemcached-awesome is licensed under the 3-Clause-BSD license, which can be found in the accompanying [LICENSE](./LICENSE) file. ## Contributing Please report any issues on the [bug tracker](https://github.com/awesomized/libmemcached/issues). A list of known permanent issues is maintained in [BUGS](./BUGS.md). All forms of contribution are welcome! Please see the bundled [CONTRIBUTING](./CONTRIBUTING.md) note for the general principles followed. The list of current and past maintainers and contributors is available in [AUTHORS](./AUTHORS). libmemcached-1.1.4/TODO000066400000000000000000000013131440143131000145670ustar00rootroot00000000000000- Stats: - redo - Deprecations: - UDP, disabled by default from memcached-1.5.6 soft (maybe only discourage in docs) - API: - memcached_delete(): expires param is obsolete Includes: - Rename all headers with a standard name like `string.h` and `limits.h`. ## Legacy: - Write a shell application (?) - Fix version in command line tools - Write (more) test cases for all command line tools (!!!) - Write some sort of "default" options bit for tools - More examples using libraries - Doxygen? - implement more connection/hash algo - implement compression - Revisit get() code (look for performance enhancements) - Add support for managing servers for clusters. - Build embedded version libmemcached-1.1.4/codecov.yml000066400000000000000000000006341440143131000162510ustar00rootroot00000000000000 ignore: - "/Applications" - "/usr" - "example" - "tests" - "test/lib/catch.hpp" fixes: - "src/libmemcachedinternal::src/libmemcached" coverage: status: project: default: target: auto threshold: 1% range: 50..80 round: nearest precision: 0 notify: gitter: default: url: "https://webhooks.gitter.im/e/a6210fd7fb7b5105b2b0" threshold: 1% libmemcached-1.1.4/contrib/000077500000000000000000000000001440143131000155415ustar00rootroot00000000000000libmemcached-1.1.4/contrib/CMakeLists.txt000066400000000000000000000000261440143131000202770ustar00rootroot00000000000000add_subdirectory(bin) libmemcached-1.1.4/contrib/bin/000077500000000000000000000000001440143131000163115ustar00rootroot00000000000000libmemcached-1.1.4/contrib/bin/CMakeLists.txt000066400000000000000000000000331440143131000210450ustar00rootroot00000000000000add_subdirectory(memaslap) libmemcached-1.1.4/contrib/bin/memaslap/000077500000000000000000000000001440143131000201105ustar00rootroot00000000000000libmemcached-1.1.4/contrib/bin/memaslap/CMakeLists.txt000066400000000000000000000023301440143131000226460ustar00rootroot00000000000000if(ENABLE_MEMASLAP AND CMAKE_USE_PTHREADS_INIT) include(CheckAtomics) check_type(cpu_set_t sched.h) check_symbol(getline stdio.h) check_symbol(_SC_NPROCESSORS_ONLN unistd.h) check_dependency(LIBEVENT event) if(HAVE_LIBEVENT AND HAVE_ATOMICS) add_executable(aslap ms_main.c ms_conn.c ms_setting.c ms_sigsegv.c ms_stats.c ms_task.c ms_thread.c) target_include_directories(aslap PRIVATE ${CMAKE_SOURCE_DIR}/include ${CMAKE_BINARY_DIR}/include ${CMAKE_SOURCE_DIR}/src ${CMAKE_BINARY_DIR}/src ${CMAKE_BINARY_DIR}) target_link_libraries(aslap PRIVATE libmemcached Threads::Threads ${LIBEVENT} m) set_target_properties(aslap PROPERTIES C_STANDARD 11 OUTPUT_NAME ${CLIENT_PREFIX}aslap) if(CMAKE_INSTALL_RPATH) set_target_properties(aslap PROPERTIES INSTALL_RPATH ${CMAKE_INSTALL_RPATH}/../${CMAKE_INSTALL_LIBDIR}) endif() install(TARGETS aslap RUNTIME COMPONENT bin DESTINATION ${CMAKE_INSTALL_BINDIR}) endif() endif() libmemcached-1.1.4/contrib/bin/memaslap/ms_atomic.h000066400000000000000000000046541440143131000222450ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef CLIENTS_MS_ATOMIC_H #define CLIENTS_MS_ATOMIC_H #include "mem_config.h" #if defined(__SUNPRO_C) || defined(HAVE_ATOMIC_ADD_NV) # define ATOMIC volatile # define _KERNEL # include # if SIZEOF_SIZE_T == 8 # define atomic_add_size(X, Y) atomic_add_64((X), (Y)) # else # define atomic_add_size(X, Y) atomic_add_32((X), (Y)) # endif # undef _KERNEL #elif HAVE_C_STDATOMIC # define ATOMIC _Atomic # include # define atomic_dec_32(X) atomic_fetch_sub(X,1) # define atomic_add_32 atomic_fetch_add # define atomic_add_32_nv(X,Y) (atomic_fetch_add(X,Y)+(Y)) # define atomic_add_size atomic_fetch_add #elif HAVE_BUILTIN_ATOMIC # define ATOMIC # define atomic_dec_32(X) BUILTIN_ATOMIC_PREFIX ## _atomic_fetch_sub(X,1) # define atomic_add_32 BUILTIN_ATOMIC_PREFIX ## _atomic_fetch_add # define atomic_add_32_nv BUILTIN_ATOMIC_PREFIX ## _atomic_add_fetch(X,Y) # define atomic_add_size BUILTIN_ATOMIC_PREFIX ## _atomic_fetch_add #elif HAVE_BUILTIN_SYNC # define ATOMIC # define atomic_dec_32(X) __sync_fetch_and_sub((X), 1) # define atomic_add_32 __sync_fetch_and_add # define atomic_add_32_nv __sync_add_and_fetch # define atomic_add_size __sync_fetch_and_add #else # warning "Atomic operators not found so memslap will not work correctly" # define ATOMIC # define atomic_dec_32(X) (--(*(X))) # define atomic_add_32(X,Y) ((*(X))+=(Y)) # define atomic_add_32_nv atomic_add_32 # define atomic_add_size atomic_add_32 #endif #endif /* CLIENTS_MS_ATOMIC_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_conn.c000066400000000000000000002340721440143131000217200ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #include #include #include #include #include #include #include #include #if defined(HAVE_ARPA_INET_H) # include #endif #if defined(HAVE_SYS_TIME_H) # include #endif #include #include "ms_setting.h" #include "ms_thread.h" #include "ms_atomic.h" #ifdef linux /* /usr/include/netinet/in.h defines macros from ntohs() to _bswap_nn to * optimize the conversion functions, but the prototypes generate warnings * from gcc. The conversion methods isn't the bottleneck for my app, so * just remove the warnings by undef'ing the optimization .. */ # undef ntohs # undef ntohl # undef htons # undef htonl #endif /* for network write */ #define TRANSMIT_COMPLETE 0 #define TRANSMIT_INCOMPLETE 1 #define TRANSMIT_SOFT_ERROR 2 #define TRANSMIT_HARD_ERROR 3 /* for generating key */ #define KEY_PREFIX_BASE 0x1010101010101010 /* not include ' ' '\r' '\n' '\0' */ #define KEY_PREFIX_MASK 0x1010101010101010 /* For parse the value length return by server */ #define KEY_TOKEN 1 #define VALUELEN_TOKEN 3 /* global increasing counter, to ensure the key prefix unique */ static uint64_t key_prefix_seq = KEY_PREFIX_BASE; /* global increasing counter, generating request id for UDP */ static ATOMIC uint32_t udp_request_id = 0; extern pthread_key_t ms_thread_key; /* generate upd request id */ static uint32_t ms_get_udp_request_id(void); /* connect initialize */ static void ms_task_init(ms_conn_t *c); static int ms_conn_udp_init(ms_conn_t *c, const bool is_udp); static int ms_conn_sock_init(ms_conn_t *c); static int ms_conn_event_init(ms_conn_t *c); static int ms_conn_init(ms_conn_t *c, const int init_state, const int read_buffer_size, const bool is_udp); static void ms_warmup_num_init(ms_conn_t *c); static int ms_item_win_init(ms_conn_t *c); /* connection close */ void ms_conn_free(ms_conn_t *c); static void ms_conn_close(ms_conn_t *c); /* create network connection */ static int ms_new_socket(struct addrinfo *ai); static void ms_maximize_sndbuf(const int sfd); static int ms_network_connect(ms_conn_t *c, char *srv_host_name, const int srv_port, const bool is_udp, int *ret_sfd); static int ms_reconn(ms_conn_t *c); /* read and parse */ static int ms_tokenize_command(char *command, token_t *tokens, const int max_tokens); static int ms_ascii_process_line(ms_conn_t *c, char *command); static int ms_try_read_line(ms_conn_t *c); static int ms_sort_udp_packet(ms_conn_t *c, char *buf, int rbytes); static int ms_udp_read(ms_conn_t *c, char *buf, int len); static int ms_try_read_network(ms_conn_t *c); static void ms_verify_value(ms_conn_t *c, ms_mlget_task_item_t *mlget_item, char *value, int vlen); static void ms_ascii_complete_nread(ms_conn_t *c); static void ms_bin_complete_nread(ms_conn_t *c); static void ms_complete_nread(ms_conn_t *c); /* send functions */ static int ms_add_msghdr(ms_conn_t *c); static int ms_ensure_iov_space(ms_conn_t *c); static int ms_add_iov(ms_conn_t *c, const void *buf, int len); static int ms_build_udp_headers(ms_conn_t *c); static int ms_transmit(ms_conn_t *c); /* status adjustment */ static void ms_conn_shrink(ms_conn_t *c); static void ms_conn_set_state(ms_conn_t *c, int state); static bool ms_update_event(ms_conn_t *c, const int new_flags); static uint32_t ms_get_rep_sock_index(ms_conn_t *c, int cmd); static uint32_t ms_get_next_sock_index(ms_conn_t *c); static int ms_update_conn_sock_event(ms_conn_t *c); static bool ms_need_yield(ms_conn_t *c); static void ms_update_start_time(ms_conn_t *c); /* main loop */ static void ms_drive_machine(ms_conn_t *c); void ms_event_handler(const int fd, const short which, void *arg); /* ascii protocol */ static int ms_build_ascii_write_buf_set(ms_conn_t *c, ms_task_item_t *item); static int ms_build_ascii_write_buf_get(ms_conn_t *c, ms_task_item_t *item); static int ms_build_ascii_write_buf_mlget(ms_conn_t *c); /* binary protocol */ static int ms_bin_process_response(ms_conn_t *c); static void ms_add_bin_header(ms_conn_t *c, uint8_t opcode, uint8_t hdr_len, uint16_t key_len, uint32_t body_len); static void ms_add_key_to_iov(ms_conn_t *c, ms_task_item_t *item); static int ms_build_bin_write_buf_set(ms_conn_t *c, ms_task_item_t *item); static int ms_build_bin_write_buf_get(ms_conn_t *c, ms_task_item_t *item); static int ms_build_bin_write_buf_mlget(ms_conn_t *c); /** * each key has two parts, prefix and suffix. The suffix is a * string random get form the character table. The prefix is a * uint64_t variable. And the prefix must be unique. we use the * prefix to identify a key. And the prefix can't include * character ' ' '\r' '\n' '\0'. * * @return uint64_t */ uint64_t ms_get_key_prefix(void) { uint64_t key_prefix; pthread_mutex_lock(&ms_global.seq_mutex); key_prefix_seq |= KEY_PREFIX_MASK; key_prefix = key_prefix_seq; key_prefix_seq++; pthread_mutex_unlock(&ms_global.seq_mutex); return key_prefix; } /* ms_get_key_prefix */ /** * get an unique udp request id * * @return an unique UDP request id */ static uint32_t ms_get_udp_request_id(void) { return atomic_add_32_nv(&udp_request_id, 1); } /** * initialize current task structure * * @param c, pointer of the concurrency */ static void ms_task_init(ms_conn_t *c) { c->curr_task.cmd = CMD_NULL; c->curr_task.item = 0; c->curr_task.verify = false; c->curr_task.finish_verify = true; c->curr_task.get_miss = true; c->curr_task.get_opt = 0; c->curr_task.set_opt = 0; c->curr_task.cycle_undo_get = 0; c->curr_task.cycle_undo_set = 0; c->curr_task.verified_get = 0; c->curr_task.overwrite_set = 0; } /* ms_task_init */ /** * initialize udp for the connection structure * * @param c, pointer of the concurrency * @param is_udp, whether it's udp * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_conn_udp_init(ms_conn_t *c, const bool is_udp) { c->hdrbuf = 0; c->rudpbuf = 0; c->udppkt = 0; c->rudpsize = UDP_DATA_BUFFER_SIZE; c->hdrsize = 0; c->rudpbytes = 0; c->packets = 0; c->recvpkt = 0; c->pktcurr = 0; c->ordcurr = 0; c->udp = is_udp; if (c->udp || (!c->udp && ms_setting.facebook_test)) { c->rudpbuf = (char *) malloc((size_t) c->rudpsize); c->udppkt = (ms_udppkt_t *) malloc(MAX_UDP_PACKET * sizeof(ms_udppkt_t)); if ((c->rudpbuf == NULL) || (c->udppkt == NULL)) { if (c->rudpbuf) free(c->rudpbuf); if (c->udppkt) free(c->udppkt); fprintf(stderr, "malloc()\n"); return -1; } memset(c->udppkt, 0, MAX_UDP_PACKET * sizeof(ms_udppkt_t)); } return EXIT_SUCCESS; } /* ms_conn_udp_init */ /** * initialize the connection structure * * @param c, pointer of the concurrency * @param init_state, (conn_read, conn_write, conn_closing) * @param read_buffer_size * @param is_udp, whether it's udp * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_conn_init(ms_conn_t *c, const int init_state, const int read_buffer_size, const bool is_udp) { assert(c); c->rbuf = c->wbuf = 0; c->iov = 0; c->msglist = 0; c->rsize = read_buffer_size; c->wsize = WRITE_BUFFER_SIZE; c->iovsize = IOV_LIST_INITIAL; c->msgsize = MSG_LIST_INITIAL; /* for replication, each connection need connect all the server */ if (ms_setting.rep_write_srv > 0) { c->total_sfds = ms_setting.srv_cnt * ms_setting.sock_per_conn; } else { c->total_sfds = ms_setting.sock_per_conn; } c->alive_sfds = 0; c->rbuf = (char *) malloc((size_t) c->rsize); c->wbuf = (char *) malloc((size_t) c->wsize); c->iov = (struct iovec *) malloc(sizeof(struct iovec) * (size_t) c->iovsize); c->msglist = (struct msghdr *) malloc(sizeof(struct msghdr) * (size_t) c->msgsize); if (ms_setting.mult_key_num > 1) { c->mlget_task.mlget_item = (ms_mlget_task_item_t *) malloc(sizeof(ms_mlget_task_item_t) * (size_t) ms_setting.mult_key_num); } c->tcpsfd = (int *) malloc((size_t) c->total_sfds * sizeof(int)); if ((c->rbuf == NULL) || (c->wbuf == NULL) || (c->iov == NULL) || (c->msglist == NULL) || (c->tcpsfd == NULL) || ((ms_setting.mult_key_num > 1) && (c->mlget_task.mlget_item == NULL))) { if (c->rbuf) free(c->rbuf); if (c->wbuf) free(c->wbuf); if (c->iov) free(c->iov); if (c->msglist) free(c->msglist); if (c->mlget_task.mlget_item) free(c->mlget_task.mlget_item); if (c->tcpsfd) free(c->tcpsfd); fprintf(stderr, "malloc()\n"); return -1; } c->state = init_state; c->rvbytes = 0; c->rbytes = 0; c->rcurr = c->rbuf; c->wcurr = c->wbuf; c->iovused = 0; c->msgcurr = 0; c->msgused = 0; c->cur_idx = c->total_sfds; /* default index is a invalid value */ c->ctnwrite = false; c->readval = false; c->change_sfd = false; c->precmd.cmd = c->currcmd.cmd = CMD_NULL; c->precmd.isfinish = true; /* default the previous command finished */ c->currcmd.isfinish = false; c->precmd.retstat = c->currcmd.retstat = MCD_FAILURE; c->precmd.key_prefix = c->currcmd.key_prefix = 0; c->mlget_task.mlget_num = 0; c->mlget_task.value_index = -1; /* default invalid value */ if (ms_setting.binary_prot_) { c->protocol = binary_prot; } else { c->protocol = ascii_prot; } /* initialize udp */ if (ms_conn_udp_init(c, is_udp)) { return -1; } /* initialize task */ ms_task_init(c); if (!(ms_setting.facebook_test && is_udp)) { atomic_add_32(&ms_stats.active_conns, 1); } return EXIT_SUCCESS; } /* ms_conn_init */ /** * when doing 100% get operation, it could preset some objects * to warmup the server. this function is used to initialize the * number of the objects to preset. * * @param c, pointer of the concurrency */ static void ms_warmup_num_init(ms_conn_t *c) { /* no set operation, preset all the items in the window */ if (ms_setting.cmd_distr[CMD_SET].cmd_prop < PROP_ERROR) { c->warmup_num = c->win_size; c->remain_warmup_num = c->warmup_num; } else { c->warmup_num = 0; c->remain_warmup_num = c->warmup_num; } } /* ms_warmup_num_init */ /** * each connection has an item window, this function initialize * the window. The window is used to generate task. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_item_win_init(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); int exp_cnt = 0; c->win_size = (int) ms_setting.win_size; c->set_cursor = 0; c->exec_num = ms_thread->thread_ctx->exec_num_perconn; c->remain_exec_num = c->exec_num; c->item_win = (ms_task_item_t *) malloc(sizeof(ms_task_item_t) * (size_t) c->win_size); if (c->item_win == NULL) { fprintf(stderr, "Can't allocate task item array for conn.\n"); return -1; } memset(c->item_win, 0, sizeof(ms_task_item_t) * (size_t) c->win_size); for (int i = 0; i < c->win_size; i++) { c->item_win[i].key_size = (int) ms_setting.distr[i].key_size; c->item_win[i].key_prefix = ms_get_key_prefix(); c->item_win[i].key_suffix_offset = ms_setting.distr[i].key_offset; c->item_win[i].value_size = (int) ms_setting.distr[i].value_size; c->item_win[i].value_offset = INVALID_OFFSET; /* default in invalid offset */ c->item_win[i].client_time = 0; /* set expire time base on the proportion */ if (exp_cnt < ms_setting.exp_ver_per * i) { c->item_win[i].exp_time = FIXED_EXPIRE_TIME; exp_cnt++; } else { c->item_win[i].exp_time = 0; } } ms_warmup_num_init(c); return EXIT_SUCCESS; } /* ms_item_win_init */ /** * each connection structure can include one or more sock * handlers. this function create these socks and connect the * server(s). * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_conn_sock_init(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); uint32_t i; int ret_sfd; uint32_t srv_idx = 0; assert(c); assert(c->tcpsfd); for (i = 0; i < c->total_sfds; i++) { ret_sfd = 0; if (ms_setting.rep_write_srv > 0) { /* for replication, each connection need connect all the server */ srv_idx = i % ms_setting.srv_cnt; } else { /* all the connections in a thread connects the same server */ srv_idx = ms_thread->thread_ctx->srv_idx; } if (ms_network_connect(c, ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port, ms_setting.udp, &ret_sfd) ) { break; } if (i == 0) { c->sfd = ret_sfd; } if (!ms_setting.udp) { c->tcpsfd[i] = ret_sfd; } c->alive_sfds++; } /* initialize udp sock handler if necessary */ if (ms_setting.facebook_test) { ret_sfd = 0; if (ms_network_connect(c, ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port, true, &ret_sfd) ) { c->udpsfd = 0; } else { c->udpsfd = ret_sfd; } } if ((i != c->total_sfds) || (ms_setting.facebook_test && (c->udpsfd == 0))) { if (ms_setting.udp) { close(c->sfd); } else { for (uint32_t j = 0; j < i; j++) { close(c->tcpsfd[j]); } } if (c->udpsfd) { close(c->udpsfd); } return -1; } return EXIT_SUCCESS; } /* ms_conn_sock_init */ /** * each connection is managed by libevent, this function * initialize the event of the connection structure. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_conn_event_init(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); short event_flags = EV_WRITE | EV_PERSIST; event_set(&c->event, c->sfd, event_flags, ms_event_handler, (void *) c); event_base_set(ms_thread->base, &c->event); c->ev_flags = event_flags; if (event_add(&c->event, NULL) == -1) { return -1; } return EXIT_SUCCESS; } /* ms_conn_event_init */ /** * setup a connection, each connection structure of each * thread must call this function to initialize. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ int ms_setup_conn(ms_conn_t *c) { if (ms_item_win_init(c)) { return -1; } if (ms_conn_init(c, conn_write, DATA_BUFFER_SIZE, ms_setting.udp)) { return -1; } if (ms_conn_sock_init(c)) { return -1; } if (ms_conn_event_init(c)) { return -1; } return EXIT_SUCCESS; } /* ms_setup_conn */ /** * Frees a connection. * * @param c, pointer of the concurrency */ void ms_conn_free(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); if (c) { if (c->hdrbuf) free(c->hdrbuf); if (c->msglist) free(c->msglist); if (c->rbuf) free(c->rbuf); if (c->wbuf) free(c->wbuf); if (c->iov) free(c->iov); if (c->mlget_task.mlget_item) free(c->mlget_task.mlget_item); if (c->rudpbuf) free(c->rudpbuf); if (c->udppkt) free(c->udppkt); if (c->item_win) free(c->item_win); if (c->tcpsfd) free(c->tcpsfd); if (--ms_thread->nactive_conn == 0) { free(ms_thread->conn); } } } /* ms_conn_free */ /** * close a connection * * @param c, pointer of the concurrency */ static void ms_conn_close(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); assert(c); /* delete the event, the socket and the connection */ event_del(&c->event); for (uint32_t i = 0; i < c->total_sfds; i++) { if (c->tcpsfd[i] > 0) { close(c->tcpsfd[i]); } } c->sfd = 0; if (ms_setting.facebook_test) { close(c->udpsfd); } atomic_dec_32(&ms_stats.active_conns); ms_conn_free(c); if (ms_setting.run_time == 0) { pthread_mutex_lock(&ms_global.run_lock.lock); ms_global.run_lock.count++; pthread_cond_signal(&ms_global.run_lock.cond); pthread_mutex_unlock(&ms_global.run_lock.lock); } if (ms_thread->nactive_conn == 0) { event_base_loopbreak(ms_thread->base); } } /* ms_conn_close */ /** * create a new sock * * @param ai, server address information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_new_socket(struct addrinfo *ai) { int sfd; if ((sfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol)) == -1) { fprintf(stderr, "socket() error: %s.\n", strerror(errno)); return -1; } return sfd; } /* ms_new_socket */ /** * Sets a socket's send buffer size to the maximum allowed by the system. * * @param sfd, file descriptor of socket */ static void ms_maximize_sndbuf(const int sfd) { socklen_t intsize = sizeof(int); unsigned int last_good = 0; unsigned int min, max, avg; unsigned int old_size; /* Start with the default size. */ if (getsockopt(sfd, SOL_SOCKET, SO_SNDBUF, &old_size, &intsize)) { fprintf(stderr, "getsockopt(SO_SNDBUF)\n"); return; } /* Binary-search for the real maximum. */ min = old_size; max = MAX_SENDBUF_SIZE; while (min <= max) { avg = ((unsigned int) (min + max)) / 2; if (setsockopt(sfd, SOL_SOCKET, SO_SNDBUF, (void *) &avg, intsize) == 0) { last_good = avg; min = avg + 1; } else { max = avg - 1; } } (void) last_good; } /* ms_maximize_sndbuf */ /** * socket connects the server * * @param c, pointer of the concurrency * @param srv_host_name, the host name of the server * @param srv_port, port of server * @param is_udp, whether it's udp * @param ret_sfd, the connected socket file descriptor * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_network_connect(ms_conn_t *c, char *srv_host_name, const int srv_port, const bool is_udp, int *ret_sfd) { int sfd; struct linger ling = {0, 0}; struct addrinfo *ai; struct addrinfo *next; struct addrinfo hints; char port_buf[NI_MAXSERV]; int error; int success = 0; int flags = 1; /* * the memset call clears nonstandard fields in some impementations * that otherwise mess things up. */ memset(&hints, 0, sizeof(hints)); #ifdef AI_ADDRCONFIG hints.ai_flags = AI_PASSIVE | AI_ADDRCONFIG; #else hints.ai_flags = AI_PASSIVE; #endif /* AI_ADDRCONFIG */ if (is_udp) { hints.ai_protocol = IPPROTO_UDP; hints.ai_socktype = SOCK_DGRAM; hints.ai_family = AF_INET; /* This left here because of issues with OSX 10.5 */ } else { hints.ai_family = AF_UNSPEC; hints.ai_protocol = IPPROTO_TCP; hints.ai_socktype = SOCK_STREAM; } snprintf(port_buf, NI_MAXSERV, "%d", srv_port); error = getaddrinfo(srv_host_name, port_buf, &hints, &ai); if (error) { if (error != EAI_SYSTEM) fprintf(stderr, "getaddrinfo(): %s.\n", gai_strerror(error)); else perror("getaddrinfo()"); return -1; } for (next = ai; next; next = next->ai_next) { if ((sfd = ms_new_socket(next)) == -1) { freeaddrinfo(ai); return -1; } setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, (void *) &flags, sizeof(flags)); if (is_udp) { ms_maximize_sndbuf(sfd); } else { setsockopt(sfd, SOL_SOCKET, SO_KEEPALIVE, (void *) &flags, sizeof(flags)); setsockopt(sfd, SOL_SOCKET, SO_LINGER, (void *) &ling, sizeof(ling)); setsockopt(sfd, IPPROTO_TCP, TCP_NODELAY, (void *) &flags, sizeof(flags)); } if (is_udp) { c->srv_recv_addr_size = sizeof(struct sockaddr); memcpy(&c->srv_recv_addr, next->ai_addr, c->srv_recv_addr_size); } else { if (connect(sfd, next->ai_addr, next->ai_addrlen) == -1) { close(sfd); freeaddrinfo(ai); return -1; } } if (((flags = fcntl(sfd, F_GETFL, 0)) < 0) || (fcntl(sfd, F_SETFL, flags | O_NONBLOCK) < 0)) { fprintf(stderr, "setting O_NONBLOCK\n"); close(sfd); freeaddrinfo(ai); return -1; } if (ret_sfd) { *ret_sfd = sfd; } success++; } freeaddrinfo(ai); /* Return zero if we detected no errors in starting up connections */ return success == 0; } /* ms_network_connect */ /** * reconnect a disconnected sock * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_reconn(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); uint32_t srv_idx = 0; uint32_t srv_conn_cnt = 0; if (ms_setting.rep_write_srv > 0) { srv_idx = c->cur_idx % ms_setting.srv_cnt; srv_conn_cnt = ms_setting.sock_per_conn * ms_setting.nconns; } else { srv_idx = ms_thread->thread_ctx->srv_idx; srv_conn_cnt = ms_setting.nconns / ms_setting.srv_cnt; } /* close the old socket handler */ close(c->sfd); c->tcpsfd[c->cur_idx] = 0; if (atomic_add_32_nv(&ms_setting.servers[srv_idx].disconn_cnt, 1) % srv_conn_cnt == 0) { gettimeofday(&ms_setting.servers[srv_idx].disconn_time, NULL); fprintf(stderr, "Server %s:%d disconnect\n", ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port); } if (ms_setting.rep_write_srv > 0) { uint32_t i = 0; for (i = 0; i < c->total_sfds; i++) { if (c->tcpsfd[i]) { break; } } /* all socks disconnect */ if (i == c->total_sfds) { return -1; } } else { do { /* reconnect success, break the loop */ if (ms_network_connect(c, ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port, ms_setting.udp, &c->sfd) == 0) { c->tcpsfd[c->cur_idx] = c->sfd; if (atomic_add_32_nv(&ms_setting.servers[srv_idx].reconn_cnt, 1) % (uint32_t) srv_conn_cnt == 0) { gettimeofday(&ms_setting.servers[srv_idx].reconn_time, NULL); int reconn_time = (int) (ms_setting.servers[srv_idx].reconn_time.tv_sec - ms_setting.servers[srv_idx].disconn_time.tv_sec); fprintf(stderr, "Server %s:%d reconnect after %ds\n", ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port, reconn_time); } break; } if (ms_setting.rep_write_srv == 0 && c->total_sfds > 0) { /* wait a second and reconnect */ sleep(1); } } while (ms_setting.rep_write_srv == 0 && c->total_sfds > 0); } if ((c->total_sfds > 1) && (c->tcpsfd[c->cur_idx] == 0)) { c->sfd = 0; c->alive_sfds--; } return EXIT_SUCCESS; } /* ms_reconn */ /** * reconnect several disconnected socks in the connection * structure, the ever-1-second timer of the thread will check * whether some socks in the connections disconnect. if * disconnect, reconnect the sock. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ int ms_reconn_socks(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); uint32_t srv_idx = 0; int ret_sfd = 0; uint32_t srv_conn_cnt = 0; struct timeval cur_time; assert(c); if ((c->total_sfds == 1) || (c->total_sfds == c->alive_sfds)) { return EXIT_SUCCESS; } for (uint32_t i = 0; i < c->total_sfds; i++) { if (c->tcpsfd[i] == 0) { gettimeofday(&cur_time, NULL); /** * For failover test of replication, reconnect the socks after * it disconnects more than 5 seconds, Otherwise memslap will * block at connect() function and the work threads can't work * in this interval. */ if (cur_time.tv_sec - ms_setting.servers[srv_idx].disconn_time.tv_sec < 5) { break; } if (ms_setting.rep_write_srv > 0) { srv_idx = i % ms_setting.srv_cnt; srv_conn_cnt = ms_setting.sock_per_conn * ms_setting.nconns; } else { srv_idx = ms_thread->thread_ctx->srv_idx; srv_conn_cnt = ms_setting.nconns / ms_setting.srv_cnt; } if (ms_network_connect(c, ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port, ms_setting.udp, &ret_sfd) == 0) { c->tcpsfd[i] = ret_sfd; c->alive_sfds++; if (atomic_add_32_nv(&ms_setting.servers[srv_idx].reconn_cnt, 1) % (uint32_t) srv_conn_cnt == 0) { gettimeofday(&ms_setting.servers[srv_idx].reconn_time, NULL); int reconn_time = (int) (ms_setting.servers[srv_idx].reconn_time.tv_sec - ms_setting.servers[srv_idx].disconn_time.tv_sec); fprintf(stderr, "Server %s:%d reconnect after %ds\n", ms_setting.servers[srv_idx].srv_host_name, ms_setting.servers[srv_idx].srv_port, reconn_time); } } } } return EXIT_SUCCESS; } /* ms_reconn_socks */ /** * Tokenize the command string by replacing whitespace with '\0' and update * the token array tokens with pointer to start of each token and length. * Returns total number of tokens. The last valid token is the terminal * token (value points to the first unprocessed character of the string and * length zero). * * Usage example: * * while(ms_tokenize_command(command, ncommand, tokens, max_tokens) > 0) { * for(int ix = 0; tokens[ix].length; ix++) { * ... * } * ncommand = tokens[ix].value - command; * command = tokens[ix].value; * } * * @param command, the command string to token * @param tokens, array to store tokens * @param max_tokens, maximum tokens number * * @return int, the number of tokens */ static int ms_tokenize_command(char *command, token_t *tokens, const int max_tokens) { char *s, *e; int ntokens = 0; assert(command && tokens && max_tokens > 1); for (s = e = command; ntokens < max_tokens - 1; ++e) { if (*e == ' ') { if (s != e) { tokens[ntokens].value = s; tokens[ntokens].length = (size_t)(e - s); ntokens++; *e = '\0'; } s = e + 1; } else if (*e == '\0') { if (s != e) { tokens[ntokens].value = s; tokens[ntokens].length = (size_t)(e - s); ntokens++; } break; /* string end */ } } return ntokens; } /* ms_tokenize_command */ /** * parse the response of server. * * @param c, pointer of the concurrency * @param command, the string responded by server * * @return int, if the command completed return EXIT_SUCCESS, else return * -1 */ static int ms_ascii_process_line(ms_conn_t *c, char *command) { int ret = 0; int64_t value_len; char *buffer = command; assert(c); /** * for command get, we store the returned value into local buffer * then continue in ms_complete_nread(). */ switch (buffer[0]) { case 'V': /* VALUE || VERSION */ if (buffer[1] == 'A') /* VALUE */ { token_t tokens[MAX_TOKENS]; ms_tokenize_command(command, tokens, MAX_TOKENS); errno = 0; value_len = strtol(tokens[VALUELEN_TOKEN].value, NULL, 10); if (errno) { printf("<%d ERROR %s\n", c->sfd, strerror(errno)); } memcpy(&c->currcmd.key_prefix, tokens[KEY_TOKEN].value, sizeof(c->currcmd.key_prefix)); /* * We read the \r\n into the string since not doing so is more * cycles then the waster of memory to do so. * * We are null terminating through, which will most likely make * some people lazy about using the return length. */ c->rvbytes = (int) (value_len + 2); c->readval = true; ret = -1; } break; case 'O': /* OK */ c->currcmd.retstat = MCD_SUCCESS; break; case 'S': /* STORED STATS SERVER_ERROR */ if (buffer[2] == 'A') /* STORED STATS */ { /* STATS*/ c->currcmd.retstat = MCD_STAT; } else if (buffer[1] == 'E') { /* SERVER_ERROR */ printf("<%d %s\n", c->sfd, buffer); c->currcmd.retstat = MCD_SERVER_ERROR; } else if (buffer[1] == 'T') { /* STORED */ c->currcmd.retstat = MCD_STORED; } else { c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; } break; case 'D': /* DELETED DATA */ if (buffer[1] == 'E') { c->currcmd.retstat = MCD_DELETED; } else { c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; } break; case 'N': /* NOT_FOUND NOT_STORED*/ if (buffer[4] == 'F') { c->currcmd.retstat = MCD_NOTFOUND; } else if (buffer[4] == 'S') { printf("<%d %s\n", c->sfd, buffer); c->currcmd.retstat = MCD_NOTSTORED; } else { c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; } break; case 'E': /* PROTOCOL ERROR or END */ if (buffer[1] == 'N') { /* END */ c->currcmd.retstat = MCD_END; } else if (buffer[1] == 'R') { printf("<%d ERROR\n", c->sfd); c->currcmd.retstat = MCD_PROTOCOL_ERROR; } else if (buffer[1] == 'X') { c->currcmd.retstat = MCD_DATA_EXISTS; printf("<%d %s\n", c->sfd, buffer); } else { c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; } break; case 'C': /* CLIENT ERROR */ printf("<%d %s\n", c->sfd, buffer); c->currcmd.retstat = MCD_CLIENT_ERROR; break; default: c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; break; } /* switch */ return ret; } /* ms_ascii_process_line */ /** * after one operation completes, reset the concurrency * * @param c, pointer of the concurrency * @param timeout, whether it's timeout */ void ms_reset_conn(ms_conn_t *c, bool timeout) { assert(c); if (c->udp) { if ((c->packets > 0) && (c->packets < MAX_UDP_PACKET)) { memset(c->udppkt, 0, sizeof(ms_udppkt_t) * (size_t) c->packets); } c->packets = 0; c->recvpkt = 0; c->pktcurr = 0; c->ordcurr = 0; c->rudpbytes = 0; } c->currcmd.isfinish = true; c->ctnwrite = false; c->rbytes = 0; c->rcurr = c->rbuf; c->msgcurr = 0; c->msgused = 0; c->iovused = 0; ms_conn_set_state(c, conn_write); memcpy(&c->precmd, &c->currcmd, sizeof(ms_cmdstat_t)); /* replicate command state */ if (timeout) { ms_drive_machine(c); } } /* ms_reset_conn */ /** * if we have a complete line in the buffer, process it. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_try_read_line(ms_conn_t *c) { if (c->protocol == binary_prot) { /* Do we have the complete packet header? */ if ((uint64_t) c->rbytes < sizeof(c->binary_header)) { /* need more data! */ return EXIT_SUCCESS; } else { #ifdef NEED_ALIGN if (((long) (c->rcurr)) % 8) { /* must realign input buffer */ memmove(c->rbuf, c->rcurr, c->rbytes); c->rcurr = c->rbuf; if (settings.verbose) { fprintf(stderr, "%d: Realign input buffer.\n", c->sfd); } } #endif protocol_binary_response_header *rsp; rsp = (protocol_binary_response_header *) c->rcurr; c->binary_header = *rsp; c->binary_header.response.extlen = rsp->response.extlen; c->binary_header.response.keylen = ntohs(rsp->response.keylen); c->binary_header.response.bodylen = ntohl(rsp->response.bodylen); c->binary_header.response.status = ntohs(rsp->response.status); if (c->binary_header.response.magic != PROTOCOL_BINARY_RES) { fprintf(stderr, "Invalid magic: %x\n", c->binary_header.response.magic); ms_conn_set_state(c, conn_closing); return EXIT_SUCCESS; } /* process this complete response */ if (ms_bin_process_response(c) == 0) { /* current operation completed */ ms_reset_conn(c, false); return -1; } else { c->rbytes -= (int32_t) sizeof(c->binary_header); c->rcurr += sizeof(c->binary_header); } } } else { char *el, *cont; assert(c); assert(c->rcurr <= (c->rbuf + c->rsize)); if (c->rbytes == 0) return EXIT_SUCCESS; el = memchr(c->rcurr, '\n', (size_t) c->rbytes); if (!el) return EXIT_SUCCESS; cont = el + 1; if (((el - c->rcurr) > 1) && (*(el - 1) == '\r')) { el--; } *el = '\0'; assert(cont <= (c->rcurr + c->rbytes)); /* process this complete line */ if (ms_ascii_process_line(c, c->rcurr) == 0) { /* current operation completed */ ms_reset_conn(c, false); return -1; } else { /* current operation didn't complete */ c->rbytes -= (int32_t)(cont - c->rcurr); c->rcurr = cont; } assert(c->rcurr <= (c->rbuf + c->rsize)); } return -1; } /* ms_try_read_line */ /** * because the packet of UDP can't ensure the order, the * function is used to sort the received udp packet. * * @param c, pointer of the concurrency * @param buf, the buffer to store the ordered packages data * @param rbytes, the maximum capacity of the buffer * * @return int, if success, return the copy bytes, else return * -1 */ static int ms_sort_udp_packet(ms_conn_t *c, char *buf, int rbytes) { int len = 0; int wbytes = 0; uint16_t req_id = 0; uint16_t seq_num = 0; uint16_t packets = 0; unsigned char *header = NULL; /* no enough data */ assert(c); assert(buf); assert(c->rudpbytes >= UDP_HEADER_SIZE); /* calculate received packets count */ if (c->rudpbytes % UDP_MAX_PAYLOAD_SIZE >= UDP_HEADER_SIZE) { /* the last packet has some data */ c->recvpkt = c->rudpbytes / UDP_MAX_PAYLOAD_SIZE + 1; } else { c->recvpkt = c->rudpbytes / UDP_MAX_PAYLOAD_SIZE; } /* get the total packets count if necessary */ if (c->packets == 0) { c->packets = HEADER_TO_PACKETS((unsigned char *) c->rudpbuf); } /* build the ordered packet array */ for (int i = c->pktcurr; i < c->recvpkt; i++) { header = (unsigned char *) c->rudpbuf + i * UDP_MAX_PAYLOAD_SIZE; req_id = (uint16_t) HEADER_TO_REQID(header); assert(req_id == c->request_id % (1 << 16)); packets = (uint16_t) HEADER_TO_PACKETS(header); assert(c->packets == HEADER_TO_PACKETS(header)); seq_num = (uint16_t) HEADER_TO_SEQNUM(header); c->udppkt[seq_num].header = header; c->udppkt[seq_num].data = (char *) header + UDP_HEADER_SIZE; if (i == c->recvpkt - 1) { /* last received packet */ if (c->rudpbytes % UDP_MAX_PAYLOAD_SIZE == 0) { c->udppkt[seq_num].rbytes = UDP_MAX_PAYLOAD_SIZE - UDP_HEADER_SIZE; c->pktcurr++; } else { c->udppkt[seq_num].rbytes = c->rudpbytes % UDP_MAX_PAYLOAD_SIZE - UDP_HEADER_SIZE; } } else { c->udppkt[seq_num].rbytes = UDP_MAX_PAYLOAD_SIZE - UDP_HEADER_SIZE; c->pktcurr++; } } for (int i = c->ordcurr; i < c->recvpkt; i++) { /* there is some data to copy */ if ((c->udppkt[i].data) && (c->udppkt[i].copybytes < c->udppkt[i].rbytes)) { header = c->udppkt[i].header; len = c->udppkt[i].rbytes - c->udppkt[i].copybytes; if (len > rbytes - wbytes) { len = rbytes - wbytes; } assert(len <= rbytes - wbytes); assert(i == HEADER_TO_SEQNUM(header)); memcpy(buf + wbytes, c->udppkt[i].data + c->udppkt[i].copybytes, (size_t) len); wbytes += len; c->udppkt[i].copybytes += len; if ((c->udppkt[i].copybytes == c->udppkt[i].rbytes) && (c->udppkt[i].rbytes == UDP_MAX_PAYLOAD_SIZE - UDP_HEADER_SIZE)) { /* finish copying all the data of this packet, next */ c->ordcurr++; } /* last received packet, and finish copying all the data */ if ((c->recvpkt == c->packets) && (i == c->recvpkt - 1) && (c->udppkt[i].copybytes == c->udppkt[i].rbytes)) { break; } /* no space to copy data */ if (wbytes >= rbytes) { break; } /* it doesn't finish reading all the data of the packet from network */ if ((i != c->recvpkt - 1) && (c->udppkt[i].rbytes < UDP_MAX_PAYLOAD_SIZE - UDP_HEADER_SIZE)) { break; } } else { /* no data to copy */ break; } } (void) packets; return wbytes == 0 ? -1 : wbytes; } /* ms_sort_udp_packet */ /** * encapsulate upd read like tcp read * * @param c, pointer of the concurrency * @param buf, read buffer * @param len, length to read * * @return int, if success, return the read bytes, else return * -1 */ static int ms_udp_read(ms_conn_t *c, char *buf, int len) { int res = 0; int avail = 0; int rbytes = 0; int copybytes = 0; assert(c->udp); while (1) { if (c->rudpbytes + UDP_MAX_PAYLOAD_SIZE > c->rudpsize) { char *new_rbuf = realloc(c->rudpbuf, (size_t) c->rudpsize * 2); if (!new_rbuf) { fprintf(stderr, "Couldn't realloc input buffer.\n"); c->rudpbytes = 0; /* ignore what we read */ return -1; } c->rudpbuf = new_rbuf; c->rudpsize *= 2; } avail = c->rudpsize - c->rudpbytes; /* UDP each time read a packet, 1400 bytes */ res = (int) read(c->sfd, c->rudpbuf + c->rudpbytes, (size_t) avail); if (res > 0) { atomic_add_size(&ms_stats.bytes_read, res); c->rudpbytes += res; rbytes += res; if (res == avail) { continue; } else { break; } } if (res == 0) { /* "connection" closed */ return res; } if (res == -1) { /* no data to read */ return res; } } /* copy data to read buffer */ if (rbytes > 0) { copybytes = ms_sort_udp_packet(c, buf, len); } if (copybytes == -1) { atomic_add_size(&ms_stats.pkt_disorder, 1); } return copybytes; } /* ms_udp_read */ /* * read from network as much as we can, handle buffer overflow and connection * close. * before reading, move the remaining incomplete fragment of a command * (if any) to the beginning of the buffer. * return EXIT_SUCCESS if there's nothing to read on the first read. */ /** * read from network as much as we can, handle buffer overflow and connection * close. before reading, move the remaining incomplete fragment of a command * (if any) to the beginning of the buffer. * * @param c, pointer of the concurrency * * @return int, * return EXIT_SUCCESS if there's nothing to read on the first read. * return EXIT_FAILURE if get data * return -1 if error happens */ static int ms_try_read_network(ms_conn_t *c) { int gotdata = 0; int res; int64_t avail; assert(c); if ((c->rcurr != c->rbuf) && (!c->readval || (c->rvbytes > c->rsize - (c->rcurr - c->rbuf)) || (c->readval && (c->rcurr - c->rbuf > c->rbytes)))) { if (c->rbytes) /* otherwise there's nothing to copy */ memmove(c->rbuf, c->rcurr, (size_t) c->rbytes); c->rcurr = c->rbuf; } while (1) { if (c->rbytes >= c->rsize) { char *new_rbuf = realloc(c->rbuf, (size_t) c->rsize * 2); if (!new_rbuf) { fprintf(stderr, "Couldn't realloc input buffer.\n"); c->rbytes = 0; /* ignore what we read */ return -1; } c->rcurr = c->rbuf = new_rbuf; c->rsize *= 2; } avail = c->rsize - c->rbytes - (c->rcurr - c->rbuf); if (avail == 0) { break; } if (c->udp) { res = (int32_t) ms_udp_read(c, c->rcurr + c->rbytes, (int32_t) avail); } else { res = (int) read(c->sfd, c->rcurr + c->rbytes, (size_t) avail); } if (res > 0) { if (!c->udp) { atomic_add_size(&ms_stats.bytes_read, res); } gotdata = 1; c->rbytes += res; if (res == avail) { continue; } else { break; } } if (res == 0) { /* connection closed */ ms_conn_set_state(c, conn_closing); return -1; } if (res == -1) { if ((errno == EAGAIN) || (EAGAIN != EWOULDBLOCK && errno == EWOULDBLOCK)) break; /* Should close on unhandled errors. */ ms_conn_set_state(c, conn_closing); return -1; } } return gotdata; } /* ms_try_read_network */ /** * after get the object from server, verify the value if * necessary. * * @param c, pointer of the concurrency * @param mlget_item, pointer of mulit-get task item structure * @param value, received value string * @param vlen, received value string length */ static void ms_verify_value(ms_conn_t *c, ms_mlget_task_item_t *mlget_item, char *value, int vlen) { if (c->curr_task.verify) { assert(c->curr_task.item->value_offset != INVALID_OFFSET); char *orignval = &ms_setting.char_block[c->curr_task.item->value_offset]; char *orignkey = &ms_setting.char_block[c->curr_task.item->key_suffix_offset]; /* verify expire time if necessary */ if (c->curr_task.item->exp_time > 0) { struct timeval curr_time; gettimeofday(&curr_time, NULL); /* object expired but get it now */ if (curr_time.tv_sec - c->curr_task.item->client_time > c->curr_task.item->exp_time + EXPIRE_TIME_ERROR) { atomic_add_size(&ms_stats.exp_get, 1); if (ms_setting.verbose) { char set_time[64]; char cur_time[64]; strftime(set_time, 64, "%Y-%m-%d %H:%M:%S", localtime(&c->curr_task.item->client_time)); strftime(cur_time, 64, "%Y-%m-%d %H:%M:%S", localtime(&curr_time.tv_sec)); fprintf(stderr, "\n<%d expire time verification failed, " "object expired but get it now\n" "\tkey len: %d\n" "\tkey: %" PRIx64 " %.*s\n" "\tset time: %s current time: %s " "diff time: %d expire time: %d\n" "\texpected data: \n" "\treceived data len: %d\n" "\treceived data: %.*s\n", c->sfd, c->curr_task.item->key_size, c->curr_task.item->key_prefix, c->curr_task.item->key_size - (int) KEY_PREFIX_SIZE, orignkey, set_time, cur_time, (int) (curr_time.tv_sec - c->curr_task.item->client_time), c->curr_task.item->exp_time, vlen, vlen, value); fflush(stderr); } } } else { if ((c->curr_task.item->value_size != vlen) || (memcmp(orignval, value, (size_t) vlen))) { atomic_add_size(&ms_stats.vef_failed, 1); if (ms_setting.verbose) { fprintf(stderr, "\n<%d data verification failed\n" "\tkey len: %d\n" "\tkey: %" PRIx64 " %.*s\n" "\texpected data len: %d\n" "\texpected data: %.*s\n" "\treceived data len: %d\n" "\treceived data: %.*s\n", c->sfd, c->curr_task.item->key_size, c->curr_task.item->key_prefix, c->curr_task.item->key_size - (int) KEY_PREFIX_SIZE, orignkey, c->curr_task.item->value_size, c->curr_task.item->value_size, orignval, vlen, vlen, value); fflush(stderr); } } } c->curr_task.finish_verify = true; if (mlget_item) { mlget_item->finish_verify = true; } } } /* ms_verify_value */ /** * For ASCII protocol, after store the data into the local * buffer, run this function to handle the data. * * @param c, pointer of the concurrency */ static void ms_ascii_complete_nread(ms_conn_t *c) { assert(c); assert(c->rbytes >= c->rvbytes); assert(c->protocol == ascii_prot); if (c->rvbytes > 2) { assert(c->rcurr[c->rvbytes - 1] == '\n' && c->rcurr[c->rvbytes - 2] == '\r'); } /* multi-get */ ms_mlget_task_item_t *mlget_item = NULL; if (((ms_setting.mult_key_num > 1) && (c->mlget_task.mlget_num >= ms_setting.mult_key_num)) || ((c->remain_exec_num == 0) && (c->mlget_task.mlget_num > 0))) { c->mlget_task.value_index++; mlget_item = &c->mlget_task.mlget_item[c->mlget_task.value_index]; if (mlget_item->item->key_prefix == c->currcmd.key_prefix) { c->curr_task.item = mlget_item->item; c->curr_task.verify = mlget_item->verify; c->curr_task.finish_verify = mlget_item->finish_verify; mlget_item->get_miss = false; } else { /* Try to find the task item in multi-get task array */ for (int i = 0; i < c->mlget_task.mlget_num; i++) { mlget_item = &c->mlget_task.mlget_item[i]; if (mlget_item->item->key_prefix == c->currcmd.key_prefix) { c->curr_task.item = mlget_item->item; c->curr_task.verify = mlget_item->verify; c->curr_task.finish_verify = mlget_item->finish_verify; mlget_item->get_miss = false; break; } } } } ms_verify_value(c, mlget_item, c->rcurr, c->rvbytes - 2); c->curr_task.get_miss = false; c->rbytes -= c->rvbytes; c->rcurr = c->rcurr + c->rvbytes; assert(c->rcurr <= (c->rbuf + c->rsize)); c->readval = false; c->rvbytes = 0; } /* ms_ascii_complete_nread */ /** * For binary protocol, after store the data into the local * buffer, run this function to handle the data. * * @param c, pointer of the concurrency */ static void ms_bin_complete_nread(ms_conn_t *c) { assert(c); assert(c->rbytes >= c->rvbytes); assert(c->protocol == binary_prot); int extlen = c->binary_header.response.extlen; int keylen = c->binary_header.response.keylen; uint8_t opcode = c->binary_header.response.opcode; /* not get command or not include value, just return */ if (((opcode != PROTOCOL_BINARY_CMD_GET) && (opcode != PROTOCOL_BINARY_CMD_GETQ)) || (c->rvbytes <= extlen + keylen)) { /* get miss */ if (c->binary_header.response.opcode == PROTOCOL_BINARY_CMD_GET) { c->currcmd.retstat = MCD_END; c->curr_task.get_miss = true; } c->readval = false; c->rvbytes = 0; ms_reset_conn(c, false); return; } /* multi-get */ ms_mlget_task_item_t *mlget_item = NULL; if (((ms_setting.mult_key_num > 1) && (c->mlget_task.mlget_num >= ms_setting.mult_key_num)) || ((c->remain_exec_num == 0) && (c->mlget_task.mlget_num > 0))) { c->mlget_task.value_index++; mlget_item = &c->mlget_task.mlget_item[c->mlget_task.value_index]; c->curr_task.item = mlget_item->item; c->curr_task.verify = mlget_item->verify; c->curr_task.finish_verify = mlget_item->finish_verify; mlget_item->get_miss = false; } ms_verify_value(c, mlget_item, c->rcurr + extlen + keylen, c->rvbytes - extlen - keylen); c->currcmd.retstat = MCD_END; c->curr_task.get_miss = false; c->rbytes -= c->rvbytes; c->rcurr = c->rcurr + c->rvbytes; assert(c->rcurr <= (c->rbuf + c->rsize)); c->readval = false; c->rvbytes = 0; if (ms_setting.mult_key_num > 1) { /* multi-get have check all the item */ if (c->mlget_task.value_index == c->mlget_task.mlget_num - 1) { ms_reset_conn(c, false); } } else { /* single get */ ms_reset_conn(c, false); } } /* ms_bin_complete_nread */ /** * we get here after reading the value of get commands. * * @param c, pointer of the concurrency */ static void ms_complete_nread(ms_conn_t *c) { assert(c); assert(c->rbytes >= c->rvbytes); assert(c->protocol == ascii_prot || c->protocol == binary_prot); if (c->protocol == binary_prot) { ms_bin_complete_nread(c); } else { ms_ascii_complete_nread(c); } } /* ms_complete_nread */ /** * Adds a message header to a connection. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_add_msghdr(ms_conn_t *c) { struct msghdr *msg; assert(c); if (c->msgsize == c->msgused) { msg = realloc(c->msglist, (size_t) c->msgsize * 2 * sizeof(struct msghdr)); if (!msg) return -1; c->msglist = msg; c->msgsize *= 2; } msg = c->msglist + c->msgused; /** * this wipes msg_iovlen, msg_control, msg_controllen, and * msg_flags, the last 3 of which aren't defined on solaris: */ memset(msg, 0, sizeof(struct msghdr)); msg->msg_iov = &c->iov[c->iovused]; if (c->udp && (c->srv_recv_addr_size > 0)) { msg->msg_name = &c->srv_recv_addr; msg->msg_namelen = c->srv_recv_addr_size; } c->msgbytes = 0; c->msgused++; if (c->udp) { /* Leave room for the UDP header, which we'll fill in later. */ return ms_add_iov(c, NULL, UDP_HEADER_SIZE); } return EXIT_SUCCESS; } /* ms_add_msghdr */ /** * Ensures that there is room for another structure iovec in a connection's * iov list. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_ensure_iov_space(ms_conn_t *c) { assert(c); if (c->iovused >= c->iovsize) { int i, iovnum; struct iovec *new_iov = (struct iovec *) realloc(c->iov, ((size_t) c->iovsize * 2) * sizeof(struct iovec)); if (!new_iov) return -1; c->iov = new_iov; c->iovsize *= 2; /* Point all the msghdr structures at the new list. */ for (i = 0, iovnum = 0; i < c->msgused; i++) { c->msglist[i].msg_iov = &c->iov[iovnum]; iovnum += (int) c->msglist[i].msg_iovlen; } } return EXIT_SUCCESS; } /* ms_ensure_iov_space */ /** * Adds data to the list of pending data that will be written out to a * connection. * * @param c, pointer of the concurrency * @param buf, the buffer includes data to send * @param len, the data length in the buffer * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_add_iov(ms_conn_t *c, const void *buf, int len) { struct msghdr *m; int leftover; bool limit_to_mtu; assert(c); do { m = &c->msglist[c->msgused - 1]; /* * Limit UDP packets, to UDP_MAX_PAYLOAD_SIZE bytes. */ limit_to_mtu = c->udp; #ifdef IOV_MAX /* We may need to start a new msghdr if this one is full. */ if ((m->msg_iovlen == IOV_MAX) || (limit_to_mtu && (c->msgbytes >= UDP_MAX_SEND_PAYLOAD_SIZE))) { ms_add_msghdr(c); m = &c->msglist[c->msgused - 1]; } #endif if (ms_ensure_iov_space(c)) return -1; /* If the fragment is too big to fit in the datagram, split it up */ if (limit_to_mtu && (len + c->msgbytes > UDP_MAX_SEND_PAYLOAD_SIZE)) { leftover = len + c->msgbytes - UDP_MAX_SEND_PAYLOAD_SIZE; len -= leftover; } else { leftover = 0; } m = &c->msglist[c->msgused - 1]; m->msg_iov[m->msg_iovlen].iov_base = (void *) buf; m->msg_iov[m->msg_iovlen].iov_len = (size_t) len; c->msgbytes += len; c->iovused++; m->msg_iovlen++; buf = ((char *) buf) + len; len = leftover; } while (leftover > 0); return EXIT_SUCCESS; } /* ms_add_iov */ /** * Constructs a set of UDP headers and attaches them to the outgoing messages. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_udp_headers(ms_conn_t *c) { int i; unsigned char *hdr; assert(c); c->request_id = ms_get_udp_request_id(); if (c->msgused > c->hdrsize) { void *new_hdrbuf; if (c->hdrbuf) new_hdrbuf = realloc(c->hdrbuf, (size_t) c->msgused * 2 * UDP_HEADER_SIZE); else new_hdrbuf = malloc((size_t) c->msgused * 2 * UDP_HEADER_SIZE); if (!new_hdrbuf) return -1; c->hdrbuf = (unsigned char *) new_hdrbuf; c->hdrsize = c->msgused * 2; } /* If this is a multi-packet request, drop it. */ if (c->udp && (c->msgused > 1)) { fprintf(stderr, "multi-packet request for UDP not supported.\n"); return -1; } hdr = c->hdrbuf; for (i = 0; i < c->msgused; i++) { c->msglist[i].msg_iov[0].iov_base = (void *) hdr; c->msglist[i].msg_iov[0].iov_len = UDP_HEADER_SIZE; *hdr++ = (unsigned char) (c->request_id / 256); *hdr++ = (unsigned char) (c->request_id % 256); *hdr++ = (unsigned char) (i / 256); *hdr++ = (unsigned char) (i % 256); *hdr++ = (unsigned char) (c->msgused / 256); *hdr++ = (unsigned char) (c->msgused % 256); *hdr++ = (unsigned char) 1; /* support facebook memcached */ *hdr++ = (unsigned char) 0; assert(hdr == ((unsigned char *) c->msglist[i].msg_iov[0].iov_base + UDP_HEADER_SIZE)); } return EXIT_SUCCESS; } /* ms_build_udp_headers */ /** * Transmit the next chunk of data from our list of msgbuf structures. * * @param c, pointer of the concurrency * * @return TRANSMIT_COMPLETE All done writing. * TRANSMIT_INCOMPLETE More data remaining to write. * TRANSMIT_SOFT_ERROR Can't write any more right now. * TRANSMIT_HARD_ERROR Can't write (c->state is set to conn_closing) */ static int ms_transmit(ms_conn_t *c) { assert(c); if ((c->msgcurr < c->msgused) && (c->msglist[c->msgcurr].msg_iovlen == 0)) { /* Finished writing the current msg; advance to the next. */ c->msgcurr++; } if (c->msgcurr < c->msgused) { ssize_t res; struct msghdr *m = &c->msglist[c->msgcurr]; res = sendmsg(c->sfd, m, 0); if (res > 0) { atomic_add_size(&ms_stats.bytes_written, res); /* We've written some of the data. Remove the completed * iovec entries from the list of pending writes. */ while (m->msg_iovlen > 0 && res >= (ssize_t) m->msg_iov->iov_len) { res -= (ssize_t) m->msg_iov->iov_len; m->msg_iovlen--; m->msg_iov++; } /* Might have written just part of the last iovec entry; * adjust it so the next write will do the rest. */ if (res > 0) { m->msg_iov->iov_base = (void *) ((unsigned char *) m->msg_iov->iov_base + res); m->msg_iov->iov_len -= (size_t) res; } return TRANSMIT_INCOMPLETE; } if ((res == -1) && ((errno == EAGAIN) || (EAGAIN != EWOULDBLOCK && errno == EWOULDBLOCK))) { if (!ms_update_event(c, EV_WRITE | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); return TRANSMIT_HARD_ERROR; } return TRANSMIT_SOFT_ERROR; } /* if res==0 or res==-1 and error is not EAGAIN or EWOULDBLOCK, * we have a real error, on which we close the connection */ fprintf(stderr, "Failed to write, and not due to blocking.\n"); ms_conn_set_state(c, conn_closing); return TRANSMIT_HARD_ERROR; } else { return TRANSMIT_COMPLETE; } } /* ms_transmit */ /** * Shrinks a connection's buffers if they're too big. This prevents * periodic large "mget" response from server chewing lots of client * memory. * * This should only be called in between requests since it can wipe output * buffers! * * @param c, pointer of the concurrency */ static void ms_conn_shrink(ms_conn_t *c) { assert(c); if (c->udp) return; if ((c->rsize > READ_BUFFER_HIGHWAT) && (c->rbytes < DATA_BUFFER_SIZE)) { char *newbuf; if (c->rcurr != c->rbuf) memmove(c->rbuf, c->rcurr, (size_t) c->rbytes); newbuf = (char *) realloc((void *) c->rbuf, DATA_BUFFER_SIZE); if (newbuf) { c->rbuf = newbuf; c->rsize = DATA_BUFFER_SIZE; } c->rcurr = c->rbuf; } if (c->udp && (c->rudpsize > UDP_DATA_BUFFER_HIGHWAT) && (c->rudpbytes + UDP_MAX_PAYLOAD_SIZE < UDP_DATA_BUFFER_SIZE)) { char *new_rbuf = (char *) realloc(c->rudpbuf, (size_t) c->rudpsize * 2); if (new_rbuf) { c->rudpbuf = new_rbuf; c->rudpsize = UDP_DATA_BUFFER_SIZE; } /* TODO check error condition? */ } if (c->msgsize > MSG_LIST_HIGHWAT) { struct msghdr *newbuf = (struct msghdr *) realloc((void *) c->msglist, MSG_LIST_INITIAL * sizeof(c->msglist[0])); if (newbuf) { c->msglist = newbuf; c->msgsize = MSG_LIST_INITIAL; } /* TODO check error condition? */ } if (c->iovsize > IOV_LIST_HIGHWAT) { struct iovec *newbuf = (struct iovec *) realloc((void *) c->iov, IOV_LIST_INITIAL * sizeof(c->iov[0])); if (newbuf) { c->iov = newbuf; c->iovsize = IOV_LIST_INITIAL; } /* TODO check return value */ } } /* ms_conn_shrink */ /** * Sets a connection's current state in the state machine. Any special * processing that needs to happen on certain state transitions can * happen here. * * @param c, pointer of the concurrency * @param state, connection state */ static void ms_conn_set_state(ms_conn_t *c, int state) { assert(c); if (state != c->state) { if (state == conn_read) { ms_conn_shrink(c); } c->state = state; } } /* ms_conn_set_state */ /** * update the event if socks change state. for example: when * change the listen scoket read event to sock write event, or * change socket handler, we could call this function. * * @param c, pointer of the concurrency * @param new_flags, new event flags * * @return bool, if success, return true, else return false */ static bool ms_update_event(ms_conn_t *c, const int new_flags) { assert(c); struct event_base *base = c->event.ev_base; if ((c->ev_flags == new_flags) && (ms_setting.rep_write_srv == 0) && (!ms_setting.facebook_test || (c->total_sfds == 1))) { return true; } if (event_del(&c->event) == -1) { /* try to delete the event again */ if (event_del(&c->event) == -1) { return false; } } event_set(&c->event, c->sfd, (short) new_flags, ms_event_handler, (void *) c); event_base_set(base, &c->event); c->ev_flags = (short) new_flags; if (event_add(&c->event, NULL) == -1) { return false; } return true; } /* ms_update_event */ /** * If user want to get the expected throughput, we could limit * the performance of memslap. we could give up some work and * just wait a short time. The function is used to check this * case. * * @param c, pointer of the concurrency * * @return bool, if success, return true, else return false */ static bool ms_need_yield(ms_conn_t *c) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); int64_t tps = 0; int64_t time_diff = 0; struct timeval curr_time; ms_task_t *task = &c->curr_task; if (ms_setting.expected_tps > 0) { gettimeofday(&curr_time, NULL); time_diff = ms_time_diff(&ms_thread->startup_time, &curr_time); tps = (int64_t)(((task->get_opt + task->set_opt) / (uint64_t) time_diff) * 1000000); /* current throughput is greater than expected throughput */ if (tps > ms_thread->thread_ctx->tps_perconn) { return true; } } return false; } /* ms_need_yield */ /** * used to update the start time of each operation * * @param c, pointer of the concurrency */ static void ms_update_start_time(ms_conn_t *c) { ms_task_item_t *item = c->curr_task.item; if ((ms_setting.stat_freq > 0) || c->udp || ((c->currcmd.cmd == CMD_SET) && (item->exp_time > 0))) { gettimeofday(&c->start_time, NULL); if ((c->currcmd.cmd == CMD_SET) && (item->exp_time > 0)) { /* record the current time */ item->client_time = c->start_time.tv_sec; } } } /* ms_update_start_time */ /** * run the state machine * * @param c, pointer of the concurrency */ static void ms_drive_machine(ms_conn_t *c) { bool stop = false; assert(c); while (!stop) { switch (c->state) { case conn_read: if (c->readval) { if (c->rbytes >= c->rvbytes) { ms_complete_nread(c); break; } } else { if (ms_try_read_line(c)) { break; } } if (ms_try_read_network(c)) { break; } /* doesn't read all the response data, wait event wake up */ if (!c->currcmd.isfinish) { if (!ms_update_event(c, EV_READ | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); break; } stop = true; break; } /* we have no command line and no data to read from network, next write */ ms_conn_set_state(c, conn_write); memcpy(&c->precmd, &c->currcmd, sizeof(ms_cmdstat_t)); /* replicate command state */ break; case conn_write: if (!c->ctnwrite && ms_need_yield(c)) { usleep(10); if (!ms_update_event(c, EV_WRITE | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); break; } stop = true; break; } if (!c->ctnwrite && (ms_exec_task(c))) { ms_conn_set_state(c, conn_closing); break; } /* record the start time before starting to send data if necessary */ if (!c->ctnwrite || (c->change_sfd && c->ctnwrite)) { if (c->change_sfd) { c->change_sfd = false; } ms_update_start_time(c); } /* change sfd if necessary */ if (c->change_sfd) { c->ctnwrite = true; stop = true; break; } /* execute task until nothing need be written to network */ if (!c->ctnwrite && (c->msgcurr == c->msgused)) { if (!ms_update_event(c, EV_WRITE | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); break; } stop = true; break; } switch (ms_transmit(c)) { case TRANSMIT_COMPLETE: /* we have no data to write to network, next wait repose */ if (!ms_update_event(c, EV_READ | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); c->ctnwrite = false; break; } ms_conn_set_state(c, conn_read); c->ctnwrite = false; stop = true; break; case TRANSMIT_INCOMPLETE: c->ctnwrite = true; break; /* Continue in state machine. */ case TRANSMIT_HARD_ERROR: c->ctnwrite = false; break; case TRANSMIT_SOFT_ERROR: c->ctnwrite = true; stop = true; break; default: break; } /* switch */ break; case conn_closing: /* recovery mode, need reconnect if connection close */ if (ms_setting.reconnect && (!ms_global.time_out || ((ms_setting.run_time == 0) && (c->remain_exec_num > 0)))) { if (ms_reconn(c)) { ms_conn_close(c); stop = true; break; } ms_reset_conn(c, false); if (c->total_sfds == 1) { if (!ms_update_event(c, EV_WRITE | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); break; } } break; } else { ms_conn_close(c); stop = true; break; } default: assert(0); } /* switch */ } } /* ms_drive_machine */ /** * the event handler of each thread * * @param fd, the file descriptor of socket * @param which, event flag * @param arg, argument */ void ms_event_handler(const int fd, const short which, void *arg) { ms_conn_t *c = (ms_conn_t *) arg; assert(c); c->which = which; /* sanity */ if (fd != c->sfd) { fprintf(stderr, "Catastrophic: event fd: %d doesn't match conn fd: %d\n", fd, c->sfd); ms_conn_close(c); exit(1); } assert(fd == c->sfd); ms_drive_machine(c); /* wait for next event */ } /* ms_event_handler */ /** * get the next socket descriptor index to run for replication * * @param c, pointer of the concurrency * @param cmd, command(get or set ) * * @return int, if success, return the index, else return EXIT_SUCCESS */ static uint32_t ms_get_rep_sock_index(ms_conn_t *c, int cmd) { uint32_t sock_index = 0; uint32_t i = 0; if (c->total_sfds == 1) { return EXIT_SUCCESS; } if (ms_setting.rep_write_srv == 0) { return sock_index; } do { if (cmd == CMD_SET) { for (i = 0; i < ms_setting.rep_write_srv; i++) { if (c->tcpsfd[i] > 0) { break; } } if (i == ms_setting.rep_write_srv) { /* random get one replication server to read */ sock_index = (uint32_t) random() % c->total_sfds; } else { /* random get one replication writing server to write */ sock_index = (uint32_t) random() % ms_setting.rep_write_srv; } } else if (cmd == CMD_GET) { /* random get one replication server to read */ sock_index = (uint32_t) random() % c->total_sfds; } } while (c->tcpsfd[sock_index] == 0); return sock_index; } /* ms_get_rep_sock_index */ /** * get the next socket descriptor index to run * * @param c, pointer of the concurrency * * @return int, return the index */ static uint32_t ms_get_next_sock_index(ms_conn_t *c) { uint32_t sock_index = 0; do { sock_index = (++c->cur_idx == c->total_sfds) ? 0 : c->cur_idx; } while (c->tcpsfd[sock_index] == 0); return sock_index; } /* ms_get_next_sock_index */ /** * update socket event of the connections * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_update_conn_sock_event(ms_conn_t *c) { assert(c); switch (c->currcmd.cmd) { case CMD_SET: if (ms_setting.facebook_test && c->udp) { c->sfd = c->tcpsfd[0]; c->udp = false; c->change_sfd = true; } break; case CMD_GET: if (ms_setting.facebook_test && !c->udp) { c->sfd = c->udpsfd; c->udp = true; c->change_sfd = true; } break; default: break; } /* switch */ if (!c->udp && (c->total_sfds > 1)) { if (c->cur_idx != c->total_sfds) { if (ms_setting.rep_write_srv == 0) { c->cur_idx = ms_get_next_sock_index(c); } else { c->cur_idx = ms_get_rep_sock_index(c, c->currcmd.cmd); } } else { /* must select the first sock of the connection at the beginning */ c->cur_idx = 0; } c->sfd = c->tcpsfd[c->cur_idx]; assert(c->sfd); c->change_sfd = true; } if (c->change_sfd) { if (!ms_update_event(c, EV_WRITE | EV_PERSIST)) { fprintf(stderr, "Couldn't update event.\n"); ms_conn_set_state(c, conn_closing); return -1; } } return EXIT_SUCCESS; } /* ms_update_conn_sock_event */ /** * for ASCII protocol, this function build the set command * string and send the command. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_ascii_write_buf_set(ms_conn_t *c, ms_task_item_t *item) { int value_offset; int write_len; char *buffer = c->wbuf; write_len = snprintf(buffer, c->wsize, " %u %d %d\r\n", 0, item->exp_time, item->value_size); if (write_len > c->wsize || write_len < 0) { /* ought to be always enough. just fail for simplicity */ fprintf(stderr, "output command line too long.\n"); return -1; } if (item->value_offset == INVALID_OFFSET) { value_offset = item->key_suffix_offset; } else { value_offset = item->value_offset; } if ((ms_add_iov(c, "set ", 4)) || (ms_add_iov(c, (char *) &item->key_prefix, (int) KEY_PREFIX_SIZE)) || (ms_add_iov(c, &ms_setting.char_block[item->key_suffix_offset], item->key_size - (int) KEY_PREFIX_SIZE) ) || (ms_add_iov(c, buffer, write_len)) || (ms_add_iov(c, &ms_setting.char_block[value_offset], item->value_size)) || (ms_add_iov(c, "\r\n", 2)) || (c->udp && (ms_build_udp_headers(c)))) { return -1; } return EXIT_SUCCESS; } /* ms_build_ascii_write_buf_set */ /** * used to send set command to server * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ int ms_mcd_set(ms_conn_t *c, ms_task_item_t *item) { assert(c); c->currcmd.cmd = CMD_SET; c->currcmd.isfinish = false; c->currcmd.retstat = MCD_FAILURE; if (ms_update_conn_sock_event(c)) { return -1; } c->msgcurr = 0; c->msgused = 0; c->iovused = 0; if (ms_add_msghdr(c)) { fprintf(stderr, "Out of memory preparing request."); return -1; } /* binary protocol */ if (c->protocol == binary_prot) { if (ms_build_bin_write_buf_set(c, item)) { return -1; } } else { if (ms_build_ascii_write_buf_set(c, item)) { return -1; } } atomic_add_size(&ms_stats.obj_bytes, item->key_size + item->value_size); atomic_add_size(&ms_stats.cmd_set, 1); return EXIT_SUCCESS; } /* ms_mcd_set */ /** * for ASCII protocol, this function build the get command * string and send the command. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_ascii_write_buf_get(ms_conn_t *c, ms_task_item_t *item) { if ((ms_add_iov(c, "get ", 4)) || (ms_add_iov(c, (char *) &item->key_prefix, (int) KEY_PREFIX_SIZE)) || (ms_add_iov(c, &ms_setting.char_block[item->key_suffix_offset], item->key_size - (int) KEY_PREFIX_SIZE) ) || (ms_add_iov(c, "\r\n", 2)) || (c->udp && (ms_build_udp_headers(c)))) { return -1; } return EXIT_SUCCESS; } /* ms_build_ascii_write_buf_get */ /** * used to send the get command to server * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ int ms_mcd_get(ms_conn_t *c, ms_task_item_t *item) { assert(c); c->currcmd.cmd = CMD_GET; c->currcmd.isfinish = false; c->currcmd.retstat = MCD_FAILURE; if (ms_update_conn_sock_event(c)) { return -1; } c->msgcurr = 0; c->msgused = 0; c->iovused = 0; if (ms_add_msghdr(c)) { fprintf(stderr, "Out of memory preparing request."); return -1; } /* binary protocol */ if (c->protocol == binary_prot) { if (ms_build_bin_write_buf_get(c, item)) { return -1; } } else { if (ms_build_ascii_write_buf_get(c, item)) { return -1; } } atomic_add_size(&ms_stats.cmd_get, 1); return EXIT_SUCCESS; } /* ms_mcd_get */ /** * for ASCII protocol, this function build the multi-get command * string and send the command. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_ascii_write_buf_mlget(ms_conn_t *c) { ms_task_item_t *item; if (ms_add_iov(c, "get", 3)) { return -1; } for (int i = 0; i < c->mlget_task.mlget_num; i++) { item = c->mlget_task.mlget_item[i].item; assert(item); if ((ms_add_iov(c, " ", 1)) || (ms_add_iov(c, (char *) &item->key_prefix, (int) KEY_PREFIX_SIZE)) || (ms_add_iov(c, &ms_setting.char_block[item->key_suffix_offset], item->key_size - (int) KEY_PREFIX_SIZE) )) { return -1; } } if ((ms_add_iov(c, "\r\n", 2)) || (c->udp && (ms_build_udp_headers(c)))) { return -1; } return EXIT_SUCCESS; } /* ms_build_ascii_write_buf_mlget */ /** * used to send the multi-get command to server * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ int ms_mcd_mlget(ms_conn_t *c) { ms_task_item_t *item; assert(c); assert(c->mlget_task.mlget_num >= 1); c->currcmd.cmd = CMD_GET; c->currcmd.isfinish = false; c->currcmd.retstat = MCD_FAILURE; if (ms_update_conn_sock_event(c)) { return -1; } c->msgcurr = 0; c->msgused = 0; c->iovused = 0; if (ms_add_msghdr(c)) { fprintf(stderr, "Out of memory preparing request."); return -1; } /* binary protocol */ if (c->protocol == binary_prot) { if (ms_build_bin_write_buf_mlget(c)) { return -1; } } else { if (ms_build_ascii_write_buf_mlget(c)) { return -1; } } /* decrease operation time of each item */ for (int i = 0; i < c->mlget_task.mlget_num; i++) { item = c->mlget_task.mlget_item[i].item; atomic_add_size(&ms_stats.cmd_get, 1); } (void) item; return EXIT_SUCCESS; } /* ms_mcd_mlget */ /** * binary protocol support */ /** * for binary protocol, parse the response of server * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_bin_process_response(ms_conn_t *c) { const char *errstr = NULL; assert(c); uint32_t bodylen = c->binary_header.response.bodylen; uint8_t opcode = c->binary_header.response.opcode; uint16_t status = c->binary_header.response.status; if (bodylen > 0) { c->rvbytes = (int32_t) bodylen; c->readval = true; return EXIT_FAILURE; } else { switch (status) { case PROTOCOL_BINARY_RESPONSE_SUCCESS: if (opcode == PROTOCOL_BINARY_CMD_SET) { c->currcmd.retstat = MCD_STORED; } else if (opcode == PROTOCOL_BINARY_CMD_DELETE) { c->currcmd.retstat = MCD_DELETED; } else if (opcode == PROTOCOL_BINARY_CMD_GET) { c->currcmd.retstat = MCD_END; } break; case PROTOCOL_BINARY_RESPONSE_ENOMEM: errstr = "Out of memory"; c->currcmd.retstat = MCD_SERVER_ERROR; break; case PROTOCOL_BINARY_RESPONSE_UNKNOWN_COMMAND: errstr = "Unknown command"; c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; break; case PROTOCOL_BINARY_RESPONSE_KEY_ENOENT: errstr = "Not found"; c->currcmd.retstat = MCD_NOTFOUND; break; case PROTOCOL_BINARY_RESPONSE_EINVAL: errstr = "Invalid arguments"; c->currcmd.retstat = MCD_PROTOCOL_ERROR; break; case PROTOCOL_BINARY_RESPONSE_KEY_EEXISTS: errstr = "Data exists for key."; break; case PROTOCOL_BINARY_RESPONSE_E2BIG: errstr = "Too large."; c->currcmd.retstat = MCD_SERVER_ERROR; break; case PROTOCOL_BINARY_RESPONSE_NOT_STORED: errstr = "Not stored."; c->currcmd.retstat = MCD_NOTSTORED; break; default: errstr = "Unknown error"; c->currcmd.retstat = MCD_UNKNOWN_READ_FAILURE; break; } /* switch */ if (errstr) { fprintf(stderr, "%s\n", errstr); } } return EXIT_SUCCESS; } /* ms_bin_process_response */ /* build binary header and add the header to the buffer to send */ /** * build binary header and add the header to the buffer to send * * @param c, pointer of the concurrency * @param opcode, operation code * @param hdr_len, length of header * @param key_len, length of key * @param body_len. length of body */ static void ms_add_bin_header(ms_conn_t *c, uint8_t opcode, uint8_t hdr_len, uint16_t key_len, uint32_t body_len) { protocol_binary_request_header *header; assert(c); header = (protocol_binary_request_header *) c->wcurr; header->request.magic = (uint8_t) PROTOCOL_BINARY_REQ; header->request.opcode = (uint8_t) opcode; header->request.keylen = htons(key_len); header->request.extlen = (uint8_t) hdr_len; header->request.datatype = (uint8_t) PROTOCOL_BINARY_RAW_BYTES; header->request.vbucket = 0; header->request.bodylen = htonl(body_len); header->request.opaque = 0; header->request.cas = 0; ms_add_iov(c, c->wcurr, sizeof(header->request)); } /* ms_add_bin_header */ /** * add the key to the socket write buffer array * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information */ static void ms_add_key_to_iov(ms_conn_t *c, ms_task_item_t *item) { ms_add_iov(c, (char *) &item->key_prefix, (int) KEY_PREFIX_SIZE); ms_add_iov(c, &ms_setting.char_block[item->key_suffix_offset], item->key_size - (int) KEY_PREFIX_SIZE); } /** * for binary protocol, this function build the set command * and add the command to send buffer array. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_bin_write_buf_set(ms_conn_t *c, ms_task_item_t *item) { assert(c->wbuf == c->wcurr); int value_offset; protocol_binary_request_set *rep = (protocol_binary_request_set *) c->wcurr; uint16_t keylen = (uint16_t) item->key_size; uint32_t bodylen = (uint32_t) sizeof(rep->message.body) + (uint32_t) keylen + (uint32_t) item->value_size; ms_add_bin_header(c, PROTOCOL_BINARY_CMD_SET, sizeof(rep->message.body), keylen, bodylen); rep->message.body.flags = 0; rep->message.body.expiration = htonl((uint32_t) item->exp_time); ms_add_iov(c, &rep->message.body, sizeof(rep->message.body)); ms_add_key_to_iov(c, item); if (item->value_offset == INVALID_OFFSET) { value_offset = item->key_suffix_offset; } else { value_offset = item->value_offset; } ms_add_iov(c, &ms_setting.char_block[value_offset], item->value_size); return EXIT_SUCCESS; } /* ms_build_bin_write_buf_set */ /** * for binary protocol, this function build the get command and * add the command to send buffer array. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_bin_write_buf_get(ms_conn_t *c, ms_task_item_t *item) { assert(c->wbuf == c->wcurr); ms_add_bin_header(c, PROTOCOL_BINARY_CMD_GET, 0, (uint16_t) item->key_size, (uint32_t) item->key_size); ms_add_key_to_iov(c, item); return EXIT_SUCCESS; } /* ms_build_bin_write_buf_get */ /** * for binary protocol, this function build the multi-get * command and add the command to send buffer array. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_build_bin_write_buf_mlget(ms_conn_t *c) { ms_task_item_t *item; assert(c->wbuf == c->wcurr); for (int i = 0; i < c->mlget_task.mlget_num; i++) { item = c->mlget_task.mlget_item[i].item; assert(item); ms_add_bin_header(c, PROTOCOL_BINARY_CMD_GET, 0, (uint16_t) item->key_size, (uint32_t) item->key_size); ms_add_key_to_iov(c, item); c->wcurr += sizeof(protocol_binary_request_get); } c->wcurr = c->wbuf; return EXIT_SUCCESS; } /* ms_build_bin_write_buf_mlget */ libmemcached-1.1.4/contrib/bin/memaslap/ms_conn.h000066400000000000000000000223011440143131000217130ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_CONN_H #define MS_CONN_H #include #include #include #include #include "ms_task.h" #include "libmemcachedprotocol-0.0/binary.h" #ifdef __cplusplus extern "C" { #endif #define DATA_BUFFER_SIZE \ (1024 * 1024 + 2048) /* read buffer, 1M + 2k, enough for the max value(1M) */ #define WRITE_BUFFER_SIZE (32 * 1024) /* write buffer, 32k */ #define UDP_DATA_BUFFER_SIZE (1 * 1024 * 1024) /* read buffer for UDP, 1M */ #define UDP_MAX_PAYLOAD_SIZE 1400 /* server limit UDP payload size */ #define UDP_MAX_SEND_PAYLOAD_SIZE 1400 /* mtu size is 1500 */ #define UDP_HEADER_SIZE 8 /* UDP header size */ #define MAX_SENDBUF_SIZE (256 * 1024 * 1024) /* Maximum socket buffer size */ #define SOCK_WAIT_TIMEOUT 30 /* maximum waiting time of UDP, 30s */ #define MAX_UDP_PACKET (1 << 16) /* maximum UDP packets, 65536 */ /* Initial size of the sendmsg() scatter/gather array. */ #define IOV_LIST_INITIAL 400 /* Initial number of sendmsg() argument structures to allocate. */ #define MSG_LIST_INITIAL 10 /* High water marks for buffer shrinking */ #define READ_BUFFER_HIGHWAT (2 * DATA_BUFFER_SIZE) #define UDP_DATA_BUFFER_HIGHWAT (4 * UDP_DATA_BUFFER_SIZE) #define IOV_LIST_HIGHWAT 600 #define MSG_LIST_HIGHWAT 100 /* parse udp header */ #define HEADER_TO_REQID(ptr) ((uint16_t) *ptr * 256 + (uint16_t) * (ptr + 1)) #define HEADER_TO_SEQNUM(ptr) ((uint16_t) * (ptr + 2) * 256 + (uint16_t) * (ptr + 3)) #define HEADER_TO_PACKETS(ptr) ((uint16_t) * (ptr + 4) * 256 + (uint16_t) * (ptr + 5)) /* states of connection */ enum conn_states { conn_read, /* reading in a command line */ conn_write, /* writing out a simple response */ conn_closing /* closing this connection */ }; /* returned states of memcached command */ enum mcd_ret { MCD_SUCCESS, /* command success */ MCD_FAILURE, /* command failure */ MCD_UNKNOWN_READ_FAILURE, /* unknown read failure */ MCD_PROTOCOL_ERROR, /* protocol error */ MCD_CLIENT_ERROR, /* client error, wrong command */ MCD_SERVER_ERROR, /* server error, server run command failed */ MCD_DATA_EXISTS, /* object is existent in server */ MCD_NOTSTORED, /* server doesn't set the object successfully */ MCD_STORED, /* server set the object successfully */ MCD_NOTFOUND, /* server not find the object */ MCD_END, /* end of the response of get command */ MCD_DELETED, /* server delete the object successfully */ MCD_STAT /* response of stats command */ }; /* used to store the current or previous running command state */ typedef struct cmdstat { int cmd; /* command name */ int retstat; /* return state of this command */ bool isfinish; /* if it read all the response data */ uint64_t key_prefix; /* key prefix */ } ms_cmdstat_t; /* udp packet structure */ typedef struct udppkt { uint8_t *header; /* udp header of the packet */ char *data; /* udp data of the packet */ int rbytes; /* number of data in the packet */ int copybytes; /* number of copied data in the packet */ } ms_udppkt_t; /* three protocols supported */ enum protocol { ascii_prot = 3, /* ASCII protocol */ binary_prot /* binary protocol */ }; /** * concurrency structure * * Each thread has a libevent to manage the events of network. * Each thread has one or more self-governed concurrencies; * each concurrency has one or more socket connections. This * concurrency structure includes all the private variables of * the concurrency. */ typedef struct conn { uint32_t conn_idx; /* connection index in the thread */ int sfd; /* current tcp sock handler of the connection structure */ int udpsfd; /* current udp sock handler of the connection structure*/ int state; /* state of the connection */ struct event event; /* event for libevent */ short ev_flags; /* event flag for libevent */ short which; /* which events were just triggered */ bool change_sfd; /* whether change sfd */ int *tcpsfd; /* TCP sock array */ uint32_t total_sfds; /* how many socks in the tcpsfd array */ uint32_t alive_sfds; /* alive socks */ uint32_t cur_idx; /* current sock index in tcpsfd array */ ms_cmdstat_t precmd; /* previous command state */ ms_cmdstat_t currcmd; /* current command state */ char *rbuf; /* buffer to read commands into */ char *rcurr; /* but if we parsed some already, this is where we stopped */ int rsize; /* total allocated size of rbuf */ int rbytes; /* how much data, starting from rcur, do we have unparsed */ bool readval; /* read value state, read known data size */ int rvbytes; /* total value size need to read */ char *wbuf; /* buffer to write commands out */ char *wcurr; /* for multi-get, where we stopped */ int wsize; /* total allocated size of wbuf */ bool ctnwrite; /* continue to write */ /* data for the mwrite state */ struct iovec *iov; int iovsize; /* number of elements allocated in iov[] */ int iovused; /* number of elements used in iov[] */ struct msghdr *msglist; int msgsize; /* number of elements allocated in msglist[] */ int msgused; /* number of elements used in msglist[] */ int msgcurr; /* element in msglist[] being transmitted now */ int msgbytes; /* number of bytes in current msg */ /* data for UDP clients */ bool udp; /* is this is a UDP "connection" */ uint32_t request_id; /* UDP request ID of current operation, if this is a UDP "connection" */ uint8_t *hdrbuf; /* udp packet headers */ int hdrsize; /* number of headers' worth of space is allocated */ struct sockaddr srv_recv_addr; /* Sent the most recent request to which server */ socklen_t srv_recv_addr_size; /* udp read buffer */ char *rudpbuf; /* buffer to read commands into for udp */ int rudpsize; /* total allocated size of rudpbuf */ int rudpbytes; /* how much data, starting from rudpbuf */ /* order udp packet */ ms_udppkt_t *udppkt; /* the offset of udp packet in rudpbuf */ int packets; /* number of total packets need to read */ int recvpkt; /* number of received packets */ int pktcurr; /* current packet in rudpbuf being ordered */ int ordcurr; /* current ordered packet */ ms_task_item_t *item_win; /* task sequence */ int win_size; /* current task window size */ uint64_t set_cursor; /* current set item index in the item window */ ms_task_t curr_task; /* current running task */ ms_mlget_task_t mlget_task; /* multi-get task */ int warmup_num; /* to run how many warm up operations*/ int remain_warmup_num; /* left how many warm up operations to run */ int64_t exec_num; /* to run how many task operations */ int64_t remain_exec_num; /* how many remained task operations to run */ /* response time statistic and time out control */ struct timeval start_time; /* start time of current operation(s) */ struct timeval end_time; /* end time of current operation(s) */ /* Binary protocol stuff */ protocol_binary_response_header binary_header; /* local temporary binary header */ enum protocol protocol; /* which protocol this connection speaks */ } ms_conn_t; /* used to generate the key prefix */ uint64_t ms_get_key_prefix(void); /** * setup a connection, each connection structure of each * thread must call this function to initialize. */ int ms_setup_conn(ms_conn_t *c); /* after one operation completes, reset the connection */ void ms_reset_conn(ms_conn_t *c, bool timeout); /** * reconnect several disconnected socks in the connection * structure, the ever-1-second timer of the thread will check * whether some socks in the connections disconnect. if * disconnect, reconnect the sock. */ int ms_reconn_socks(ms_conn_t *c); /* used to send set command to server */ int ms_mcd_set(ms_conn_t *c, ms_task_item_t *item); /* used to send the get command to server */ int ms_mcd_get(ms_conn_t *c, ms_task_item_t *item); /* used to send the multi-get command to server */ int ms_mcd_mlget(ms_conn_t *c); #ifdef __cplusplus } #endif #endif /* end of MS_CONN_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_main.c000066400000000000000000000614451440143131000217110ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #include #include #include #include #include #if defined(HAVE_SYS_TIME_H) # include #endif #include #include "ms_atomic.h" #include "ms_sigsegv.h" #include "ms_setting.h" #include "ms_thread.h" /* global structure */ ms_global_t ms_global; /* global stats information structure */ ms_stats_t ms_stats; /* global statistic structure */ ms_statistic_t ms_statistic; #define PROGRAM_NAME "memaslap" #define PROGRAM_DESCRIPTION "Generates workload against memcached servers." #ifdef __sun /* For some odd reason the option struct on solaris defines the argument * as char* and not const char* */ # define OPTIONSTRING char * #else # define OPTIONSTRING const char * #endif /* options */ static struct option long_options[] = { {(OPTIONSTRING) "servers", required_argument, NULL, OPT_SERVERS}, {(OPTIONSTRING) "threads", required_argument, NULL, OPT_THREAD_NUMBER}, {(OPTIONSTRING) "concurrency", required_argument, NULL, OPT_CONCURRENCY}, {(OPTIONSTRING) "conn_sock", required_argument, NULL, OPT_SOCK_PER_CONN}, {(OPTIONSTRING) "execute_number", required_argument, NULL, OPT_EXECUTE_NUMBER}, {(OPTIONSTRING) "time", required_argument, NULL, OPT_TIME}, {(OPTIONSTRING) "cfg_cmd", required_argument, NULL, OPT_CONFIG_CMD}, {(OPTIONSTRING) "win_size", required_argument, NULL, OPT_WINDOW_SIZE}, {(OPTIONSTRING) "fixed_size", required_argument, NULL, OPT_FIXED_LTH}, {(OPTIONSTRING) "verify", required_argument, NULL, OPT_VERIFY}, {(OPTIONSTRING) "division", required_argument, NULL, OPT_GETS_DIVISION}, {(OPTIONSTRING) "stat_freq", required_argument, NULL, OPT_STAT_FREQ}, {(OPTIONSTRING) "exp_verify", required_argument, NULL, OPT_EXPIRE}, {(OPTIONSTRING) "overwrite", required_argument, NULL, OPT_OVERWRITE}, {(OPTIONSTRING) "reconnect", no_argument, NULL, OPT_RECONNECT}, {(OPTIONSTRING) "udp", no_argument, NULL, OPT_UDP}, {(OPTIONSTRING) "facebook", no_argument, NULL, OPT_FACEBOOK_TEST}, {(OPTIONSTRING) "binary", no_argument, NULL, OPT_BINARY_PROTOCOL}, {(OPTIONSTRING) "tps", required_argument, NULL, OPT_TPS}, {(OPTIONSTRING) "rep_write", required_argument, NULL, OPT_REP_WRITE_SRV}, {(OPTIONSTRING) "verbose", no_argument, NULL, OPT_VERBOSE}, {(OPTIONSTRING) "help", no_argument, NULL, OPT_HELP}, {(OPTIONSTRING) "version", no_argument, NULL, OPT_VERSION}, {0, 0, 0, 0}, }; /* Prototypes */ static void ms_sync_lock_init(void); static void ms_sync_lock_destroy(void); static void ms_global_struct_init(void); static void ms_global_struct_destroy(void); static void ms_version_command(const char *command_name); static const char *ms_lookup_help(ms_options_t option); static int64_t ms_parse_time(void); static int64_t ms_parse_size(void); static void ms_options_parse(int argc, char *argv[]); static int ms_check_para(void); static void ms_statistic_init(void); static void ms_stats_init(void); static void ms_print_statistics(int in_time); static void ms_print_memaslap_stats(struct timeval *start_time, struct timeval *end_time); static void ms_monitor_slap_mode(void); /** * output the help information * * @param command_name, the string of this process * @param description, description of this process * @param long_options, global options array */ static void ms_help_command(const char *command_name, const char *description) { char *help_message = NULL; printf("%s v%u.%u\n", command_name, 1U, 0U); printf(" %s\n\n", description); printf("Usage:\n" " memaslap -hV | -s servers [-F config_file] [-t time | -x exe_num] [...]\n\n" "Options:\n"); for (int x = 0; long_options[x].name; x++) { printf(" -%c, --%s%c\n", long_options[x].val, long_options[x].name, long_options[x].has_arg ? '=' : ' '); if ((help_message = (char *) ms_lookup_help(long_options[x].val))) { printf(" %s\n", help_message); } } printf("\nExamples:\n" " memaslap -s 127.0.0.1:11211 -S 5s\n" " memaslap -s 127.0.0.1:11211 -t 2m -v 0.2 -e 0.05 -b\n" " memaslap -s 127.0.0.1:11211 -F config -t 2m -w 40k -S 20s -o 0.2\n" " memaslap -s 127.0.0.1:11211 -F config -t 2m -T 4 -c 128 -d 20 -P 40k\n" " memaslap -s 127.0.0.1:11211 -F config -t 2m -d 50 -a -n 40\n" " memaslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m\n" " memaslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m -p 2\n\n"); } /* ms_help_command */ /* initialize the global locks */ static void ms_sync_lock_init() { ms_global.init_lock.count = 0; pthread_mutex_init(&ms_global.init_lock.lock, NULL); pthread_cond_init(&ms_global.init_lock.cond, NULL); ms_global.warmup_lock.count = 0; pthread_mutex_init(&ms_global.warmup_lock.lock, NULL); pthread_cond_init(&ms_global.warmup_lock.cond, NULL); ms_global.run_lock.count = 0; pthread_mutex_init(&ms_global.run_lock.lock, NULL); pthread_cond_init(&ms_global.run_lock.cond, NULL); pthread_mutex_init(&ms_global.quit_mutex, NULL); pthread_mutex_init(&ms_global.seq_mutex, NULL); } /* ms_sync_lock_init */ /* destroy the global locks */ static void ms_sync_lock_destroy() { pthread_mutex_destroy(&ms_global.init_lock.lock); pthread_cond_destroy(&ms_global.init_lock.cond); pthread_mutex_destroy(&ms_global.warmup_lock.lock); pthread_cond_destroy(&ms_global.warmup_lock.cond); pthread_mutex_destroy(&ms_global.run_lock.lock); pthread_cond_destroy(&ms_global.run_lock.cond); pthread_mutex_destroy(&ms_global.quit_mutex); pthread_mutex_destroy(&ms_global.seq_mutex); if (ms_setting.stat_freq > 0) { pthread_mutex_destroy(&ms_statistic.stat_mutex); } } /* ms_sync_lock_destroy */ /* initialize the global structure */ static void ms_global_struct_init() { ms_sync_lock_init(); ms_global.finish_warmup = false; ms_global.time_out = false; } /* destroy the global structure */ static void ms_global_struct_destroy() { ms_sync_lock_destroy(); } /** * output the version information * * @param command_name, the string of this process */ static void ms_version_command(const char *command_name) { printf("%s v%u.%u\n", command_name, 1U, 0U); exit(0); } /** * get the description of the option * * @param option, option of command line * * @return char*, description of the command option */ static const char *ms_lookup_help(ms_options_t option) { switch (option) { case OPT_SERVERS: return "List one or more servers to connect. Servers count must be less than\n" " threads count. e.g.: --servers=localhost:1234,localhost:11211"; case OPT_VERSION: return "Display the version of the application and then exit."; case OPT_HELP: return "Display this message and then exit."; case OPT_EXECUTE_NUMBER: return "Number of operations(get and set) to execute for the\n" " given test. Default 1000000."; case OPT_THREAD_NUMBER: return "Number of threads to startup, better equal to CPU numbers. Default 8."; case OPT_CONCURRENCY: return "Number of concurrency to simulate with load. Default 128."; case OPT_FIXED_LTH: return "Fixed length of value."; case OPT_VERIFY: return "The proportion of date verification, e.g.: --verify=0.01"; case OPT_GETS_DIVISION: return "Number of keys to multi-get once. Default 1, means single get."; case OPT_TIME: return "How long the test to run, suffix: s-seconds, m-minutes, h-hours,\n" " d-days e.g.: --time=2h."; case OPT_CONFIG_CMD: return "Load the configure file to get command,key and value distribution list."; case OPT_WINDOW_SIZE: return "Task window size of each concurrency, suffix: K, M e.g.: --win_size=10k.\n" " Default 10k."; case OPT_UDP: return "UDP support, by default memaslap uses TCP; TCP port and UDP port of\n" " server must be same."; case OPT_EXPIRE: return "The proportion of objects with expire time, e.g.: --exp_verify=0.01.\n" " Default no object with expire time"; case OPT_OVERWRITE: return "The proportion of objects need overwrite, e.g.: --overwrite=0.01.\n" " Default never overwrite object."; case OPT_STAT_FREQ: return "Frequency of dumping statistic information. suffix: s-seconds,\n" " m-minutes, e.g.: --resp_freq=10s."; case OPT_SOCK_PER_CONN: return "Number of TCP socks per concurrency. Default 1."; case OPT_RECONNECT: return "Reconnect support, when connection is closed it will be reconnected."; case OPT_VERBOSE: return "Whether it outputs detailed information when verification fails."; case OPT_FACEBOOK_TEST: return "Whether it enables facebook test feature, set with TCP and multi-get with UDP."; case OPT_BINARY_PROTOCOL: return "Whether it enables binary protocol. Default with ASCII protocol."; case OPT_TPS: return "Expected throughput, suffix: K, e.g.: --tps=10k."; case OPT_REP_WRITE_SRV: return "The first nth servers can write data, e.g.: --rep_write=2."; default: return "Forgot to document this option :)"; } /* switch */ } /* ms_lookup_help */ /* used to parse the time string */ static int64_t ms_parse_time() { int64_t ret = 0; char unit = optarg[strlen(optarg) - 1]; optarg[strlen(optarg) - 1] = '\0'; ret = atoi(optarg); switch (unit) { case 'd': case 'D': ret *= 24; /* fall through */ case 'h': case 'H': ret *= 60; /* fall through */ case 'm': case 'M': ret *= 60; /* fall through */ case 's': case 'S': break; default: ret = -1; break; } /* switch */ return ret; } /* ms_parse_time */ /* used to parse the size string */ static int64_t ms_parse_size() { int64_t ret = -1; char unit = optarg[strlen(optarg) - 1]; optarg[strlen(optarg) - 1] = '\0'; errno = 0; ret = strtoll(optarg, (char **) NULL, 10); if (errno) { fprintf(stderr, "strtoll(optarg,..): %s\n", strerror(errno)); exit(1); } switch (unit) { case 'k': case 'K': ret *= 1024; break; case 'm': case 'M': ret *= 1024 * 1024; break; case 'g': case 'G': ret *= 1024 * 1024 * 1024; break; default: ret = -1; break; } /* switch */ return ret; } /* ms_parse_size */ /* used to parse the options of command line */ static void ms_options_parse(int argc, char *argv[]) { int option_index = 0; int option_rv; while ((option_rv = getopt_long(argc, argv, "VhURbaBs:x:T:c:X:v:d:" "t:S:F:w:e:o:n:P:p:", long_options, &option_index)) != -1) { switch (option_rv) { case 0: break; case OPT_VERSION: /* --version or -V */ ms_version_command(PROGRAM_NAME); break; case OPT_HELP: /* --help or -h */ ms_help_command(PROGRAM_NAME, PROGRAM_DESCRIPTION); exit(0); break; case OPT_SERVERS: /* --servers or -s */ ms_setting.srv_str = strdup(optarg); break; case OPT_CONCURRENCY: /* --concurrency or -c */ errno = 0; ms_setting.nconns = (uint32_t) strtoul(optarg, (char **) NULL, 10); if (ms_setting.nconns <= 0 || errno) { fprintf(stderr, "Concurrency must be greater than 0.:-)\n"); exit(1); } break; case OPT_EXECUTE_NUMBER: /* --execute_number or -x */ errno = 0; ms_setting.exec_num = (int) strtol(optarg, (char **) NULL, 10); if (ms_setting.exec_num <= 0 || errno) { fprintf(stderr, "Execute number must be greater than 0.:-)\n"); exit(1); } break; case OPT_THREAD_NUMBER: /* --threads or -T */ errno = 0; ms_setting.nthreads = (uint32_t) strtoul(optarg, (char **) NULL, 10); if (ms_setting.nthreads <= 0 || errno) { fprintf(stderr, "Threads number must be greater than 0.:-)\n"); exit(1); } break; case OPT_FIXED_LTH: /* --fixed_size or -X */ errno = 0; ms_setting.fixed_value_size = (size_t) strtoull(optarg, (char **) NULL, 10); if ((ms_setting.fixed_value_size <= 0 || errno) || (ms_setting.fixed_value_size > MAX_VALUE_SIZE)) { fprintf(stderr, "Value size must be between 0 and 1M.:-)\n"); exit(1); } break; case OPT_VERIFY: /* --verify or -v */ ms_setting.verify_percent = atof(optarg); if ((ms_setting.verify_percent <= 0) || (ms_setting.verify_percent > 1.0)) { fprintf(stderr, "Data verification rate must be " "greater than 0 and less than 1.0. :-)\n"); exit(1); } break; case OPT_GETS_DIVISION: /* --division or -d */ errno = 0; ms_setting.mult_key_num = (int) strtol(optarg, (char **) NULL, 10); if (ms_setting.mult_key_num <= 0 || errno) { fprintf(stderr, "Multi-get key number must be greater than 0.:-)\n"); exit(1); } break; case OPT_TIME: /* --time or -t */ ms_setting.run_time = (int) ms_parse_time(); if (ms_setting.run_time == -1) { fprintf(stderr, "Please specify the run time. :-)\n" "'s' for second, 'm' for minute, 'h' for hour, " "'d' for day. e.g.: --time=24h (means 24 hours).\n"); exit(1); } if (ms_setting.run_time == 0) { fprintf(stderr, "Running time can not be 0. :-)\n"); exit(1); } break; case OPT_CONFIG_CMD: /* --cfg_cmd or -F */ ms_setting.cfg_file = strdup(optarg); break; case OPT_WINDOW_SIZE: /* --win_size or -w */ ms_setting.win_size = (size_t) ms_parse_size(); if (ms_setting.win_size == (size_t) -1) { fprintf(stderr, "Please specify the item window size. :-)\n" "e.g.: --win_size=10k (means 10k task window size).\n"); exit(1); } break; case OPT_UDP: /* --udp or -U*/ ms_setting.udp = true; break; case OPT_EXPIRE: /* --exp_verify or -e */ ms_setting.exp_ver_per = atof(optarg); if ((ms_setting.exp_ver_per <= 0) || (ms_setting.exp_ver_per > 1.0)) { fprintf(stderr, "Expire time verification rate must be " "greater than 0 and less than 1.0. :-)\n"); exit(1); } break; case OPT_OVERWRITE: /* --overwrite or -o */ ms_setting.overwrite_percent = atof(optarg); if ((ms_setting.overwrite_percent <= 0) || (ms_setting.overwrite_percent > 1.0)) { fprintf(stderr, "Objects overwrite rate must be " "greater than 0 and less than 1.0. :-)\n"); exit(1); } break; case OPT_STAT_FREQ: /* --stat_freq or -S */ ms_setting.stat_freq = (int) ms_parse_time(); if (ms_setting.stat_freq == -1) { fprintf(stderr, "Please specify the frequency of dumping " "statistic information. :-)\n" "'s' for second, 'm' for minute, 'h' for hour, " "'d' for day. e.g.: --time=24h (means 24 hours).\n"); exit(1); } if (ms_setting.stat_freq == 0) { fprintf(stderr, "The frequency of dumping statistic information " "can not be 0. :-)\n"); exit(1); } break; case OPT_SOCK_PER_CONN: /* --conn_sock or -n */ errno = 0; ms_setting.sock_per_conn = (uint32_t) strtoul(optarg, (char **) NULL, 10); if (ms_setting.sock_per_conn <= 0 || errno) { fprintf(stderr, "Number of socks of each concurrency " "must be greater than 0.:-)\n"); exit(1); } break; case OPT_RECONNECT: /* --reconnect or -R */ ms_setting.reconnect = true; break; case OPT_VERBOSE: /* --verbose or -b */ ms_setting.verbose = true; break; case OPT_FACEBOOK_TEST: /* --facebook or -a */ ms_setting.facebook_test = true; break; case OPT_BINARY_PROTOCOL: /* --binary or -B */ ms_setting.binary_prot_ = true; break; case OPT_TPS: /* --tps or -P */ ms_setting.expected_tps = (int) ms_parse_size(); if (ms_setting.expected_tps == -1) { fprintf(stderr, "Please specify the item expected throughput. :-)\n" "e.g.: --tps=10k (means 10k throughput).\n"); exit(1); } break; case OPT_REP_WRITE_SRV: /* --rep_write or -p */ errno = 0; ms_setting.rep_write_srv = (uint32_t) strtoul(optarg, (char **) NULL, 10); if (ms_setting.rep_write_srv <= 0 || errno) { fprintf(stderr, "Number of replication writing server must be greater " "than 0.:-)\n"); exit(1); } break; case '?': /* getopt_long already printed an error message. */ exit(1); default: abort(); } /* switch */ } } /* ms_options_parse */ static int ms_check_para() { if (ms_setting.srv_str == NULL) { char *temp; if ((temp = getenv("MEMCACHED_SERVERS"))) { ms_setting.srv_str = strdup(temp); } else { fprintf(stderr, "No servers provided\n\n"); return -1; } } if (ms_setting.nconns % (uint32_t) ms_setting.nthreads) { fprintf(stderr, "Concurrency must be the multiples of threads count.\n"); return -1; } if (ms_setting.win_size % UNIT_ITEMS_COUNT) { fprintf(stderr, "Window size must be the multiples of 1024.\n\n"); return -1; } return EXIT_SUCCESS; } /* ms_check_para */ /* initialize the statistic structure */ static void ms_statistic_init() { pthread_mutex_init(&ms_statistic.stat_mutex, NULL); ms_init_stats(&ms_statistic.get_stat, "Get"); ms_init_stats(&ms_statistic.set_stat, "Set"); ms_init_stats(&ms_statistic.total_stat, "Total"); } /* ms_statistic_init */ /* initialize the global state structure */ static void ms_stats_init() { memset(&ms_stats, 0, sizeof(ms_stats_t)); if (ms_setting.stat_freq > 0) { ms_statistic_init(); } } /* ms_stats_init */ /* use to output the statistic */ static void ms_print_statistics(int in_time) { int obj_size = (int) (ms_setting.avg_key_size + ms_setting.avg_val_size); printf("\033[1;1H\033[2J\n"); ms_dump_format_stats(&ms_statistic.get_stat, in_time, ms_setting.stat_freq, obj_size); ms_dump_format_stats(&ms_statistic.set_stat, in_time, ms_setting.stat_freq, obj_size); ms_dump_format_stats(&ms_statistic.total_stat, in_time, ms_setting.stat_freq, obj_size); } /* ms_print_statistics */ /* used to print the states of memaslap */ static void ms_print_memaslap_stats(struct timeval *start_time, struct timeval *end_time) { char buf[0x2000]; char *pos = buf; pos += snprintf(pos, sizeof(buf), "cmd_get: %lu\n", (unsigned long) ms_stats.cmd_get); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "cmd_set: %lu\n", (unsigned long) ms_stats.cmd_set); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "get_misses: %lu\n", (unsigned long) ms_stats.get_misses); if (ms_setting.verify_percent > 0) { pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "verify_misses: %lu\n", (unsigned long) ms_stats.vef_miss); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "verify_failed: %lu\n", (unsigned long) ms_stats.vef_failed); } if (ms_setting.exp_ver_per > 0) { pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "expired_get: %lu\n", (unsigned long) ms_stats.exp_get); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "unexpired_unget: %lu\n", (unsigned long) ms_stats.unexp_unget); } pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "written_bytes: %lu\n", (unsigned long) ms_stats.bytes_written); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "read_bytes: %lu\n", (unsigned long) ms_stats.bytes_read); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "object_bytes: %lu\n", (unsigned long) ms_stats.obj_bytes); if (ms_setting.udp || ms_setting.facebook_test) { pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "packet_disorder: %lu\n", (unsigned long) ms_stats.pkt_disorder); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "packet_drop: %lu\n", (unsigned long) ms_stats.pkt_drop); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "udp_timeout: %lu\n", (unsigned long) ms_stats.udp_timeout); } if (ms_setting.stat_freq > 0) { ms_dump_stats(&ms_statistic.get_stat); ms_dump_stats(&ms_statistic.set_stat); ms_dump_stats(&ms_statistic.total_stat); } int64_t time_diff = ms_time_diff(start_time, end_time); pos += snprintf(pos, sizeof(buf) - (size_t)(pos - buf), "\nRun time: %.1fs Ops: %llu TPS: %.0Lf Net_rate: %.1fM/s\n", (double) time_diff / 1000000, (unsigned long long) (ms_stats.cmd_get + ms_stats.cmd_set), (ms_stats.cmd_get + ms_stats.cmd_set) / ((long double) time_diff / 1000000), (double) (ms_stats.bytes_written + ms_stats.bytes_read) / 1024 / 1024 / ((double) time_diff / 1000000)); assert(pos <= (buf + sizeof(buf))); fwrite(buf, 1, pos - buf, stdout); fflush(stdout); } /* ms_print_memaslap_stats */ /* the loop of the main thread, wait the work threads to complete */ static void ms_monitor_slap_mode() { struct timeval start_time, end_time; /* Wait all the threads complete initialization. */ pthread_mutex_lock(&ms_global.init_lock.lock); while (ms_global.init_lock.count < ms_setting.nthreads) { pthread_cond_wait(&ms_global.init_lock.cond, &ms_global.init_lock.lock); } pthread_mutex_unlock(&ms_global.init_lock.lock); /* only when there is no set operation it need warm up */ if (ms_setting.cmd_distr[CMD_SET].cmd_prop < PROP_ERROR) { /* Wait all the connects complete warm up. */ pthread_mutex_lock(&ms_global.warmup_lock.lock); while (ms_global.warmup_lock.count < ms_setting.nconns) { pthread_cond_wait(&ms_global.warmup_lock.cond, &ms_global.warmup_lock.lock); } pthread_mutex_unlock(&ms_global.warmup_lock.lock); } ms_global.finish_warmup = true; /* running in "run time" mode, user specify run time */ if (ms_setting.run_time > 0) { int second = 0; gettimeofday(&start_time, NULL); while (1) { sleep(1); second++; if ((ms_setting.stat_freq > 0) && (second % ms_setting.stat_freq == 0) && (ms_stats.active_conns >= ms_setting.nconns) && (ms_stats.active_conns <= INT_MAX)) { ms_print_statistics(second); } if (ms_setting.run_time <= second) { ms_global.time_out = true; break; } /* all connections disconnect */ if ((second > 5) && (ms_stats.active_conns == 0)) { break; } } gettimeofday(&end_time, NULL); sleep(1); /* wait all threads clean up */ } else { /* running in "execute number" mode, user specify execute number */ gettimeofday(&start_time, NULL); /* * We loop until we know that all connects have cleaned up. */ pthread_mutex_lock(&ms_global.run_lock.lock); while (ms_global.run_lock.count < ms_setting.nconns) { pthread_cond_wait(&ms_global.run_lock.cond, &ms_global.run_lock.lock); } pthread_mutex_unlock(&ms_global.run_lock.lock); gettimeofday(&end_time, NULL); } ms_print_memaslap_stats(&start_time, &end_time); } /* ms_monitor_slap_mode */ /* the main function */ int main(int argc, char *argv[]) { srandom((unsigned int) time(NULL)); ms_global_struct_init(); /* initialization */ ms_setting_init_pre(); ms_options_parse(argc, argv); if (ms_check_para()) { ms_help_command(PROGRAM_NAME, PROGRAM_DESCRIPTION); exit(1); } ms_setting_init_post(); ms_stats_init(); ms_thread_init(); /* waiting work thread complete its task */ ms_monitor_slap_mode(); /* clean up */ ms_thread_cleanup(); ms_global_struct_destroy(); ms_setting_cleanup(); return EXIT_SUCCESS; } /* main */ libmemcached-1.1.4/contrib/bin/memaslap/ms_memslap.h000066400000000000000000000101321440143131000224130ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_MEMSLAP_H #define MS_MEMSLAP_H #include #include #include #include #include #include #include #include #if !defined(__cplusplus) # include #endif #include #include "ms_stats.h" #include "ms_atomic.h" #ifdef __cplusplus extern "C" { #endif /* command line option */ typedef enum { OPT_VERSION = 'V', OPT_HELP = 'h', OPT_UDP = 'U', OPT_SERVERS = 's', OPT_EXECUTE_NUMBER = 'x', OPT_THREAD_NUMBER = 'T', OPT_CONCURRENCY = 'c', OPT_FIXED_LTH = 'X', OPT_VERIFY = 'v', OPT_GETS_DIVISION = 'd', OPT_TIME = 't', OPT_CONFIG_CMD = 'F', OPT_WINDOW_SIZE = 'w', OPT_EXPIRE = 'e', OPT_STAT_FREQ = 'S', OPT_RECONNECT = 'R', OPT_VERBOSE = 'b', OPT_FACEBOOK_TEST = 'a', OPT_SOCK_PER_CONN = 'n', OPT_BINARY_PROTOCOL = 'B', OPT_OVERWRITE = 'o', OPT_TPS = 'P', OPT_REP_WRITE_SRV = 'p' } ms_options_t; /* global statistic of response time */ typedef struct statistic { pthread_mutex_t stat_mutex; /* synchronize the following members */ ms_stat_t get_stat; /* statistics of get command */ ms_stat_t set_stat; /* statistics of set command */ ms_stat_t total_stat; /* statistics of both get and set commands */ } ms_statistic_t; /* global status statistic structure */ typedef struct stats { ATOMIC uint32_t active_conns; /* active connections */ ATOMIC size_t bytes_read; /* read bytes */ ATOMIC size_t bytes_written; /* written bytes */ ATOMIC size_t obj_bytes; /* object bytes */ ATOMIC size_t pre_cmd_get; /* previous total get command count */ ATOMIC size_t pre_cmd_set; /* previous total set command count */ ATOMIC size_t cmd_get; /* current total get command count */ ATOMIC size_t cmd_set; /* current total set command count */ ATOMIC size_t get_misses; /* total objects of get miss */ ATOMIC size_t vef_miss; /* total objects of verification miss */ ATOMIC size_t vef_failed; /* total objects of verification failed */ ATOMIC size_t unexp_unget; /* total objects which is unexpired but not get */ ATOMIC size_t exp_get; /* total objects which is expired but get */ ATOMIC size_t pkt_disorder; /* disorder packages of UDP */ ATOMIC size_t pkt_drop; /* packages dropped of UDP */ ATOMIC size_t udp_timeout; /* how many times timeout of UDP happens */ } ms_stats_t; /* lock adapter */ typedef struct sync_lock { uint32_t count; pthread_mutex_t lock; pthread_cond_t cond; } ms_sync_lock_t; /* global variable structure */ typedef struct global { /* synchronize lock */ ms_sync_lock_t init_lock; ms_sync_lock_t warmup_lock; ms_sync_lock_t run_lock; /* mutex for outputing error log synchronously when memslap crashes */ pthread_mutex_t quit_mutex; /* mutex for generating key prefix */ pthread_mutex_t seq_mutex; /* global synchronous flags for slap mode */ ATOMIC bool finish_warmup; ATOMIC bool time_out; } ms_global_t; /* global structure */ extern ms_global_t ms_global; /* global stats information structure */ extern ms_stats_t ms_stats; /* global statistic structure */ extern ms_statistic_t ms_statistic; #ifdef __cplusplus } #endif #endif /* end of MS_MEMSLAP_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_setting.c000066400000000000000000000647561440143131000224520ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #include "libmemcached/memcached.h" #include #include #include #include #if HAVE_STRINGS_H # include #endif #include #if HAVE_UNISTD_H # include #endif #include "ms_setting.h" #include "ms_conn.h" #define MAX_EXEC_NUM 0x4000000000000000 /* 1 << 62 */ #define ADDR_ALIGN(addr) ((addr + 15) & ~(16 - 1)) /* 16 bytes aligned */ #define RAND_CHAR_SIZE (10 * 1024 * 1024) /* 10M character table */ #define RESERVED_RAND_CHAR_SIZE (2 * 1024 * 1024) /* reserved 2M to avoid pointer sloping over */ #define DEFAULT_CONFIG_NAME ".memslap.cnf" #define DEFAULT_THREADS_NUM 1 /* default start one thread */ #define DEFAULT_CONNS_NUM 16 /* default each thread with 16 connections */ #define DEFAULT_EXE_NUM 0 /* default execute number is 0 */ #define DEFAULT_VERIFY_RATE 0.0 /* default it doesn't do data verification */ #define DEFAULT_OVERWRITE_RATE 0.0 /* default it doesn't do overwrite */ #define DEFAULT_DIV 1 /* default it runs single get */ #define DEFAULT_RUN_TIME 600 /* default run time 10 minutes */ #define DEFAULT_WINDOW_SIZE (10 * UNIT_ITEMS_COUNT) /* default window size is 10k */ #define DEFAULT_SOCK_PER_CONN 1 /* default socks per connection is 1 */ /* Use this for string generation */ #define CHAR_COUNT 64 /* number of characters used to generate character table */ const char ALPHANUMBERICS[] = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz.-"; ms_setting_st ms_setting; /* store the settings specified by user */ /* read setting from configuration file */ static void ms_get_serverlist(char *str); static uint32_t ms_get_cpu_count(void); ms_conf_type_t ms_get_conf_type(char *line); static int ms_is_line_data(char *line); static int ms_read_is_data(char *line, ssize_t nread); static void ms_no_config_file(void); static void ms_parse_cfg_file(char *cfg_file); /* initialize setting structure */ static void ms_init_random_block(void); static void ms_calc_avg_size(void); static int ms_shuffle_distr(ms_distr_t *distr, int length); static void ms_build_distr(void); static void ms_print_setting(void); static void ms_setting_slapmode_init_pre(void); static void ms_setting_slapmode_init_post(void); #if !defined(HAVE_GETLINE) # include static ssize_t getline(char **line, size_t *line_size, FILE *fp) { char delim = '\n'; ssize_t result = 0; size_t cur_len = 0; if (line == NULL || line_size == NULL || fp == NULL) { errno = EINVAL; return -1; } if (*line == NULL || *line_size == 0) { char *new_line; *line_size = 120; new_line = (char *) realloc(*line, *line_size); if (new_line == NULL) { result = -1; return result; } *line = new_line; } for (;;) { int i = getc(fp); if (i == EOF) { result = -1; break; } /* Make enough space for len+1 (for final NUL) bytes. */ if (cur_len + 1 >= *line_size) { size_t needed_max = SSIZE_MAX < SIZE_MAX ? (size_t) SSIZE_MAX + 1 : SIZE_MAX; size_t needed = (2 * (*line_size)) + 1; char *new_line; if (needed_max < needed) needed = needed_max; if (cur_len + 1 >= needed) { result = -1; errno = EOVERFLOW; return result; } new_line = (char *) realloc(*line, needed); if (new_line == NULL) { result = -1; return result; } *line = new_line; *line_size = needed; } (*line)[cur_len] = (char) i; cur_len++; if (i == delim) break; } (*line)[cur_len] = '\0'; if (cur_len) return (ssize_t) cur_len; return result; } #endif /** * parse the server list string, and build the servers * information structure array. this function is used to parse * the command line options specified by user. * * @param str, the string of server list */ static void ms_get_serverlist(char *str) { ms_mcd_server_t *srvs = NULL; /** * Servers list format is like this. For example: * "localhost:11108, localhost:11109" */ memcached_server_st *server_pool; server_pool = memcached_servers_parse(str); for (uint32_t loop = 0; loop < memcached_server_list_count(server_pool); loop++) { assert(ms_setting.srv_cnt < ms_setting.total_srv_cnt); strcpy(ms_setting.servers[ms_setting.srv_cnt].srv_host_name, server_pool[loop].hostname); ms_setting.servers[ms_setting.srv_cnt].srv_port = server_pool[loop].port; ms_setting.servers[ms_setting.srv_cnt].disconn_cnt = 0; ms_setting.servers[ms_setting.srv_cnt].reconn_cnt = 0; ms_setting.srv_cnt++; if (ms_setting.srv_cnt >= ms_setting.total_srv_cnt) { srvs = (ms_mcd_server_t *) realloc( ms_setting.servers, (size_t) ms_setting.total_srv_cnt * sizeof(ms_mcd_server_t) * 2); if (srvs == NULL) { fprintf(stderr, "Can't reallocate servers structure.\n"); exit(1); } ms_setting.servers = srvs; ms_setting.total_srv_cnt *= 2; } } memcached_server_free(server_pool); } /* ms_get_serverlist */ /** * used to get the CPU count of the current system * * @return return the cpu count if possible, else return 1 */ static uint32_t ms_get_cpu_count() { #ifdef HAVE_CPU_SET_T unsigned cpu_count = 0; cpu_set_t cpu_set; sched_getaffinity(0, sizeof(cpu_set_t), &cpu_set); for (unsigned i = 0; i < (sizeof(cpu_set_t) * 8); i++) { if (CPU_ISSET(i, &cpu_set)) { cpu_count++; } } return cpu_count; #elif defined HAVE__SC_NPROCESSORS_ONLN return sysconf(_SC_NPROCESSORS_CONF); #endif /* the system with one cpu at least */ return 1; } /* ms_get_cpu_count */ /** * used to get the configure type based on the type string read * from the configuration file. * * @param line, string of one line * * @return ms_conf_type_t */ ms_conf_type_t ms_get_conf_type(char *line) { if (!memcmp(line, "key", strlen("key"))) { return CONF_KEY; } else if (!memcmp(line, "value", strlen("value"))) { return CONF_VALUE; } else if (!memcmp(line, "cmd", strlen("cmd"))) { return CONF_CMD; } else { return CONF_NULL; } } /* ms_get_conf_type */ /** * judge whether the line is a line with useful data. used to * parse the configuration file. * * @param line, string of one line * * @return if success, return EXIT_FAILURE, else return EXIT_SUCCESS */ static int ms_is_line_data(char *line) { assert(line); char *begin_ptr = line; while (isspace(*begin_ptr)) { begin_ptr++; } if ((begin_ptr[0] == '\0') || (begin_ptr[0] == '#')) return EXIT_SUCCESS; return EXIT_FAILURE; } /* ms_is_line_data */ /** * function to bypass blank line and comments * * @param line, string of one line * @param nread, length of the line * * @return if it's EOF or not line data, return EXIT_SUCCESS, else return EXIT_FAILURE */ static int ms_read_is_data(char *line, ssize_t nread) { if ((nread == EOF) || !ms_is_line_data(line)) return EXIT_SUCCESS; return EXIT_FAILURE; } /* ms_read_is_data */ /** * if no configuration file, use this function to create the default * configuration file. */ static void ms_no_config_file() { char userpath[PATH_MAX]; struct passwd *usr = NULL; FILE *fd; usr = getpwuid(getuid()); snprintf(userpath, PATH_MAX, "%s/%s", usr->pw_dir, DEFAULT_CONFIG_NAME); if (access(userpath, F_OK | R_OK) == 0) goto exit; fd = fopen(userpath, "w+"); if (fd == NULL) { fprintf(stderr, "Could not create default configure file %s\n", userpath); perror("fopen"); exit(1); } fprintf(fd, "%s", DEFAULT_CONGIF_STR); fclose(fd); exit: ms_setting.cfg_file = strdup(userpath); } /* ms_no_config_file */ /** * parse the configuration file * * @param cfg_file, the configuration file name */ static void ms_parse_cfg_file(char *cfg_file) { FILE *f; size_t start_len, end_len; double proportion; char *line = NULL; size_t read_len; ssize_t nread; int cmd_type; ms_conf_type_t conf_type; int end_of_file = 0; ms_key_distr_t *key_distr = NULL; ms_value_distr_t *val_distr = NULL; if (cfg_file == NULL) { ms_no_config_file(); cfg_file = ms_setting.cfg_file; } /*read key value configure file*/ if ((f = fopen(cfg_file, "r")) == NULL) { fprintf(stderr, "Can not open file: '%s'.\n", cfg_file); exit(1); } while (1) { if ((((nread = getline(&line, &read_len, f)) == 1) || !ms_read_is_data(line, nread)) && (nread != EOF)) /* bypass blank line */ continue; if (nread == EOF) { fprintf(stderr, "Bad configuration file, no configuration find.\n"); exit(1); } conf_type = ms_get_conf_type(line); break; } while (!end_of_file) { switch (conf_type) { case CONF_KEY: while (1) { if ((((nread = getline(&line, &read_len, f)) == 1) || !ms_read_is_data(line, nread)) && (nread != EOF)) /* bypass blank line */ continue; if (nread != EOF) { if (sscanf(line, "%zu %zu %lf ", &start_len, &end_len, &proportion) != 3) { conf_type = ms_get_conf_type(line); break; } ms_setting.key_distr[ms_setting.key_rng_cnt].start_len = start_len; ms_setting.key_distr[ms_setting.key_rng_cnt].end_len = end_len; ms_setting.key_distr[ms_setting.key_rng_cnt].key_prop = proportion; ms_setting.key_rng_cnt++; if (ms_setting.key_rng_cnt >= ms_setting.total_key_rng_cnt) { key_distr = (ms_key_distr_t *) realloc(ms_setting.key_distr, (size_t) ms_setting.total_key_rng_cnt * sizeof(ms_key_distr_t) * 2); if (key_distr == NULL) { fprintf(stderr, "Can't reallocate key distribution structure.\n"); exit(1); } ms_setting.key_distr = key_distr; ms_setting.total_key_rng_cnt *= 2; } continue; } end_of_file = 1; break; } break; case CONF_VALUE: while (1) { if ((((nread = getline(&line, &read_len, f)) == 1) || !ms_read_is_data(line, nread)) && (nread != EOF)) /* bypass blank line */ continue; if (nread != EOF) { if (sscanf(line, "%zu %zu %lf", &start_len, &end_len, &proportion) != 3) { conf_type = ms_get_conf_type(line); break; } ms_setting.value_distr[ms_setting.val_rng_cnt].start_len = start_len; ms_setting.value_distr[ms_setting.val_rng_cnt].end_len = end_len; ms_setting.value_distr[ms_setting.val_rng_cnt].value_prop = proportion; ms_setting.val_rng_cnt++; if (ms_setting.val_rng_cnt >= ms_setting.total_val_rng_cnt) { val_distr = (ms_value_distr_t *) realloc(ms_setting.value_distr, (size_t) ms_setting.total_val_rng_cnt * sizeof(ms_value_distr_t) * 2); if (val_distr == NULL) { fprintf(stderr, "Can't reallocate key distribution structure.\n"); exit(1); } ms_setting.value_distr = val_distr; ms_setting.total_val_rng_cnt *= 2; } continue; } end_of_file = 1; break; } break; case CONF_CMD: while (1) { if ((((nread = getline(&line, &read_len, f)) == 1) || !ms_read_is_data(line, nread)) && (nread != EOF)) /* bypass blank line */ continue; if (nread != EOF) { if (sscanf(line, "%d %lf", &cmd_type, &proportion) != 2) { conf_type = ms_get_conf_type(line); break; } if (cmd_type >= CMD_NULL) { continue; } ms_setting.cmd_distr[ms_setting.cmd_used_count].cmd_type = cmd_type; ms_setting.cmd_distr[ms_setting.cmd_used_count].cmd_prop = proportion; ms_setting.cmd_used_count++; continue; } end_of_file = 1; break; } case CONF_NULL: while (1) { if ((((nread = getline(&line, &read_len, f)) == 1) || !ms_read_is_data(line, nread)) && (nread != EOF)) /* bypass blank line */ continue; if (nread != EOF) { if ((conf_type = ms_get_conf_type(line)) != CONF_NULL) { break; } continue; } end_of_file = 1; break; } break; default: assert(0); break; } /* switch */ } fclose(f); if (line) { free(line); } } /* ms_parse_cfg_file */ /* calculate the average size of key and value */ static void ms_calc_avg_size() { double avg_val_size = 0.0; double avg_key_size = 0.0; double val_pro = 0.0; double key_pro = 0.0; double averge_len = 0.0; size_t start_len = 0; size_t end_len = 0; for (int j = 0; j < ms_setting.val_rng_cnt; j++) { val_pro = ms_setting.value_distr[j].value_prop; start_len = ms_setting.value_distr[j].start_len; end_len = ms_setting.value_distr[j].end_len; averge_len = val_pro * ((double) (start_len + end_len)) / 2; avg_val_size += averge_len; } for (int j = 0; j < ms_setting.key_rng_cnt; j++) { key_pro = ms_setting.key_distr[j].key_prop; start_len = ms_setting.key_distr[j].start_len; end_len = ms_setting.key_distr[j].end_len; averge_len = key_pro * ((double) (start_len + end_len)) / 2; avg_key_size += averge_len; } ms_setting.avg_val_size = (size_t) avg_val_size; ms_setting.avg_key_size = (size_t) avg_key_size; } /* ms_calc_avg_size */ /** * used to shuffle key and value distribution array to ensure * (key, value) pair with different set. * * @param distr, pointer of distribution structure array * @param length, length of the array * * @return always return EXIT_SUCCESS */ static int ms_shuffle_distr(ms_distr_t *distr, int length) { int i, j; int tmp_offset; size_t tmp_size; int64_t rnd; for (i = 0; i < length; i++) { rnd = random(); j = (int) (rnd % (length - i)) + i; switch (rnd % 3) { case 0: tmp_size = distr[j].key_size; distr[j].key_size = distr[i].key_size; distr[i].key_size = tmp_size; break; case 1: tmp_offset = distr[j].key_offset; distr[j].key_offset = distr[i].key_offset; distr[i].key_offset = tmp_offset; break; case 2: tmp_size = distr[j].value_size; distr[j].value_size = distr[i].value_size; distr[i].value_size = tmp_size; break; default: break; } /* switch */ } return EXIT_SUCCESS; } /* ms_shuffle_distr */ /** * according to the key and value distribution, to build the * (key, value) pair distribution. the (key, value) pair * distribution array is global, each connection set or get * object keeping this distribution, for the final result, we * can reach the expected key and value distribution. */ static void ms_build_distr() { int offset = 0; int end = 0; int key_cnt = 0; int value_cnt = 0; size_t average_len = 0; size_t diff_len = 0; size_t start_len = 0; size_t end_len = 0; int rnd = 0; ms_distr_t *distr = NULL; int units = (int) ms_setting.win_size / UNIT_ITEMS_COUNT; /* calculate average value size and key size */ ms_calc_avg_size(); ms_setting.char_blk_size = RAND_CHAR_SIZE; int key_scope_size = (int) ((ms_setting.char_blk_size - RESERVED_RAND_CHAR_SIZE) / UNIT_ITEMS_COUNT); ms_setting.distr = (ms_distr_t *) malloc(sizeof(ms_distr_t) * ms_setting.win_size); if (ms_setting.distr == NULL) { fprintf(stderr, "Can't allocate distribution array."); exit(1); } /** * character block is divided by how many different key * size, each different key size has the same size character * range. */ for (int m = 0; m < units; m++) { for (int i = 0; i < UNIT_ITEMS_COUNT; i++) { ms_setting.distr[m * UNIT_ITEMS_COUNT + i].key_offset = ADDR_ALIGN(key_scope_size * i); } } /* initialize key size distribution */ for (int m = 0; m < units; m++) { for (int j = 0; j < ms_setting.key_rng_cnt; j++) { key_cnt = (int) (UNIT_ITEMS_COUNT * ms_setting.key_distr[j].key_prop); start_len = ms_setting.key_distr[j].start_len; end_len = ms_setting.key_distr[j].end_len; if ((start_len < MIN_KEY_SIZE) || (end_len < MIN_KEY_SIZE)) { fprintf(stderr, "key length must be greater than 16 bytes.\n"); exit(1); } if (!ms_setting.binary_prot_ && ((start_len > MAX_KEY_SIZE) || (end_len > MAX_KEY_SIZE))) { fprintf(stderr, "key length must be less than 250 bytes.\n"); exit(1); } average_len = (start_len + end_len) / 2; diff_len = (end_len - start_len) / 2; for (int k = 0; k < key_cnt; k++) { if (offset >= (m + 1) * UNIT_ITEMS_COUNT) { break; } rnd = (int) random(); if (k % 2 == 0) { ms_setting.distr[offset].key_size = (diff_len == 0) ? average_len : average_len + (size_t) rnd % diff_len; } else { ms_setting.distr[offset].key_size = (diff_len == 0) ? average_len : average_len - (size_t) rnd % diff_len; } offset++; } } if (offset < (m + 1) * UNIT_ITEMS_COUNT) { end = (m + 1) * UNIT_ITEMS_COUNT - offset; for (int i = 0; i < end; i++) { ms_setting.distr[offset].key_size = ms_setting.avg_key_size; offset++; } } } offset = 0; /* initialize value distribution */ if (ms_setting.fixed_value_size) { for (int i = 0; i < units * UNIT_ITEMS_COUNT; i++) { ms_setting.distr[i].value_size = ms_setting.fixed_value_size; } } else { for (int m = 0; m < units; m++) { for (int j = 0; j < ms_setting.val_rng_cnt; j++) { value_cnt = (int) (UNIT_ITEMS_COUNT * ms_setting.value_distr[j].value_prop); start_len = ms_setting.value_distr[j].start_len; end_len = ms_setting.value_distr[j].end_len; if ((start_len <= 0) || (end_len <= 0)) { fprintf(stderr, "value length must be greater than 0 bytes.\n"); exit(1); } if ((start_len > MAX_VALUE_SIZE) || (end_len > MAX_VALUE_SIZE)) { fprintf(stderr, "key length must be less than or equal to 1M.\n"); exit(1); } average_len = (start_len + end_len) / 2; diff_len = (end_len - start_len) / 2; for (int k = 0; k < value_cnt; k++) { if (offset >= (m + 1) * UNIT_ITEMS_COUNT) { break; } rnd = (int) random(); if (k % 2 == 0) { ms_setting.distr[offset].value_size = (diff_len == 0) ? average_len : average_len + (size_t) rnd % diff_len; } else { ms_setting.distr[offset].value_size = (diff_len == 0) ? average_len : average_len - (size_t) rnd % diff_len; } offset++; } } if (offset < (m + 1) * UNIT_ITEMS_COUNT) { end = (m + 1) * UNIT_ITEMS_COUNT - offset; for (int i = 0; i < end; i++) { ms_setting.distr[offset++].value_size = ms_setting.avg_val_size; } } } } /* shuffle distribution */ for (int i = 0; i < units; i++) { distr = &ms_setting.distr[i * UNIT_ITEMS_COUNT]; for (int j = 0; j < 4; j++) { ms_shuffle_distr(distr, UNIT_ITEMS_COUNT); } } } /* ms_build_distr */ /** * used to initialize the global character block. The character * block is used to generate the suffix of the key and value. we * only store a pointer in the character block for each key * suffix or value string. It can save much memory to store key * or value string. */ static void ms_init_random_block() { char *ptr = NULL; assert(ms_setting.char_blk_size > 0); ms_setting.char_block = (char *) malloc(ms_setting.char_blk_size); if (ms_setting.char_block == NULL) { fprintf(stderr, "Can't allocate global char block."); exit(1); } ptr = ms_setting.char_block; for (int i = 0; (size_t) i < ms_setting.char_blk_size; i++) { *(ptr++) = ALPHANUMBERICS[random() % CHAR_COUNT]; } } /* ms_init_random_block */ /** * after initialization, call this function to output the main * configuration user specified. */ static void ms_print_setting() { fprintf(stdout, "servers: %s\n", ms_setting.srv_str); fprintf(stdout, "threads count: %d\n", ms_setting.nthreads); fprintf(stdout, "concurrency: %d\n", ms_setting.nconns); if (ms_setting.run_time > 0) { fprintf(stdout, "run time: %ds\n", ms_setting.run_time); } else { fprintf(stdout, "execute number: %" PRId64 "\n", ms_setting.exec_num); } fprintf(stdout, "windows size: %" PRId64 "k\n", (int64_t)(ms_setting.win_size / 1024)); fprintf(stdout, "set proportion: set_prop=%.2f\n", ms_setting.cmd_distr[CMD_SET].cmd_prop); fprintf(stdout, "get proportion: get_prop=%.2f\n", ms_setting.cmd_distr[CMD_GET].cmd_prop); fflush(stdout); } /* ms_print_setting */ /** * previous part of slap mode initialization of setting structure */ static void ms_setting_slapmode_init_pre() { ms_setting.exec_num = DEFAULT_EXE_NUM; ms_setting.verify_percent = DEFAULT_VERIFY_RATE; ms_setting.exp_ver_per = DEFAULT_VERIFY_RATE; ms_setting.overwrite_percent = DEFAULT_OVERWRITE_RATE; ms_setting.mult_key_num = DEFAULT_DIV; ms_setting.fixed_value_size = 0; ms_setting.win_size = DEFAULT_WINDOW_SIZE; ms_setting.udp = false; ms_setting.reconnect = false; ms_setting.verbose = false; ms_setting.facebook_test = false; ms_setting.binary_prot_ = false; ms_setting.stat_freq = 0; ms_setting.srv_str = NULL; ms_setting.cfg_file = NULL; ms_setting.sock_per_conn = DEFAULT_SOCK_PER_CONN; ms_setting.expected_tps = 0; ms_setting.rep_write_srv = 0; } /* ms_setting_slapmode_init_pre */ /** * previous part of initialization of setting structure */ void ms_setting_init_pre() { memset(&ms_setting, 0, sizeof(ms_setting)); /* common initialize */ ms_setting.ncpu = ms_get_cpu_count(); ms_setting.nthreads = DEFAULT_THREADS_NUM; ms_setting.nconns = DEFAULT_CONNS_NUM; ms_setting.run_time = DEFAULT_RUN_TIME; ms_setting.total_srv_cnt = MCD_SRVS_NUM_INIT; ms_setting.servers = (ms_mcd_server_t *) malloc((size_t) ms_setting.total_srv_cnt * sizeof(ms_mcd_server_t)); if (ms_setting.servers == NULL) { fprintf(stderr, "Can't allocate servers structure.\n"); exit(1); } ms_setting_slapmode_init_pre(); } /* ms_setting_init_pre */ /** * post part of slap mode initialization of setting structure */ static void ms_setting_slapmode_init_post() { ms_setting.total_key_rng_cnt = KEY_RANGE_COUNT_INIT; ms_setting.key_distr = (ms_key_distr_t *) malloc((size_t) ms_setting.total_key_rng_cnt * sizeof(ms_key_distr_t)); if (ms_setting.key_distr == NULL) { fprintf(stderr, "Can't allocate key distribution structure.\n"); exit(1); } ms_setting.total_val_rng_cnt = VALUE_RANGE_COUNT_INIT; ms_setting.value_distr = (ms_value_distr_t *) malloc((size_t) ms_setting.total_val_rng_cnt * sizeof(ms_value_distr_t)); if (ms_setting.value_distr == NULL) { fprintf(stderr, "Can't allocate value distribution structure.\n"); exit(1); } ms_parse_cfg_file(ms_setting.cfg_file); /* run time mode */ if ((ms_setting.exec_num == 0) && (ms_setting.run_time)) { ms_setting.exec_num = (int64_t) MAX_EXEC_NUM; } else { /* execute number mode */ ms_setting.run_time = 0; } if (ms_setting.rep_write_srv > 0) { /* for replication test, need enable reconnect feature */ ms_setting.reconnect = true; } if (ms_setting.facebook_test && (ms_setting.mult_key_num < 2)) { fprintf(stderr, "facebook test must work with multi-get, " "please specify multi-get key number " "with '--division' option.\n"); exit(1); } if (ms_setting.facebook_test && ms_setting.udp) { fprintf(stderr, "facebook test couldn't work with UDP.\n"); exit(1); } if (ms_setting.udp && (ms_setting.sock_per_conn > 1)) { fprintf(stderr, "UDP doesn't support multi-socks " "in one connection structure.\n"); exit(1); } if ((ms_setting.rep_write_srv > 0) && (ms_setting.srv_cnt < 2)) { fprintf(stderr, "Please specify 2 servers at least for replication\n"); exit(1); } if ((ms_setting.rep_write_srv > 0) && (ms_setting.srv_cnt < ms_setting.rep_write_srv)) { fprintf(stderr, "Servers to do replication writing " "is larger than the total servers\n"); exit(1); } if (ms_setting.udp && (ms_setting.rep_write_srv > 0)) { fprintf(stderr, "UDP doesn't support replication.\n"); exit(1); } if (ms_setting.facebook_test && (ms_setting.rep_write_srv > 0)) { fprintf(stderr, "facebook test couldn't work with replication.\n"); exit(1); } ms_build_distr(); /* initialize global character block */ ms_init_random_block(); ms_print_setting(); } /* ms_setting_slapmode_init_post */ /** * post part of initialization of setting structure */ void ms_setting_init_post() { ms_get_serverlist(ms_setting.srv_str); ms_setting_slapmode_init_post(); } /** * clean up the global setting structure */ void ms_setting_cleanup() { if (ms_setting.distr) { free(ms_setting.distr); } if (ms_setting.char_block) { free(ms_setting.char_block); } if (ms_setting.srv_str) { free(ms_setting.srv_str); } if (ms_setting.cfg_file) { free(ms_setting.cfg_file); } if (ms_setting.servers) { free(ms_setting.servers); } if (ms_setting.key_distr) { free(ms_setting.key_distr); } if (ms_setting.value_distr) { free(ms_setting.value_distr); } } /* ms_setting_cleanup */ libmemcached-1.1.4/contrib/bin/memaslap/ms_setting.h000066400000000000000000000142341440143131000224410ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_SETTING_H #define MS_SETTING_H #include "ms_memslap.h" #ifdef HAVE_SYS_TIME_H # include #endif #ifdef __cplusplus extern "C" { #endif #define MCD_SRVS_NUM_INIT 8 #define MCD_HOST_LENGTH 64 #define KEY_RANGE_COUNT_INIT 8 #define VALUE_RANGE_COUNT_INIT 8 #define PROP_ERROR 0.001 #define MIN_KEY_SIZE 16 #define MAX_KEY_SIZE 250 #define MAX_VALUE_SIZE (1024 * 1024) /* the content of the configuration file for memslap running without configuration file */ #define DEFAULT_CONGIF_STR \ "key\n" \ "64 64 1\n" \ "value\n" \ "1024 1024 1\n" \ "cmd\n" \ "0 0.1\n" \ "1 0.9" /* Used to parse the value length return by server and path string */ typedef struct token_s { char *value; size_t length; } token_t; #define MAX_TOKENS 10 /* server information */ typedef struct mcd_server { char srv_host_name[MCD_HOST_LENGTH]; /* host name of server */ int srv_port; /* server port */ /* for calculating how long the server disconnects */ ATOMIC uint32_t disconn_cnt; /* number of disconnections count */ ATOMIC uint32_t reconn_cnt; /* number of reconnections count */ struct timeval disconn_time; /* start time of disconnection */ struct timeval reconn_time; /* end time of reconnection */ } ms_mcd_server_t; /* information of an item distribution including key and value */ typedef struct distr { size_t key_size; /* size of key */ int key_offset; /* offset of one key in character block */ size_t value_size; /* size of value */ } ms_distr_t; /* information of key distribution */ typedef struct key_distr { size_t start_len; /* start of the key length range */ size_t end_len; /* end of the key length range */ double key_prop; /* key proportion */ } ms_key_distr_t; /* information of value distribution */ typedef struct value_distr { size_t start_len; /* start of the value length range */ size_t end_len; /* end of the value length range */ double value_prop; /* value proportion */ } ms_value_distr_t; /* memcached command types */ typedef enum cmd_type { CMD_SET, CMD_GET, CMD_NULL } ms_cmd_type_t; /* types in the configuration file */ typedef enum conf_type { CONF_KEY, CONF_VALUE, CONF_CMD, CONF_NULL } ms_conf_type_t; /* information of command distribution */ typedef struct cmd_distr { ms_cmd_type_t cmd_type; /* command type */ double cmd_prop; /* proportion of the command */ } ms_cmd_distr_t; /* global setting structure */ typedef struct setting { uint32_t ncpu; /* cpu count of this system */ uint32_t nthreads; /* total thread count, must equal or less than cpu cores */ uint32_t nconns; /* total conn count, must multiply by total thread count */ int64_t exec_num; /* total execute number */ int run_time; /* total run time */ uint32_t char_blk_size; /* global character block size */ char *char_block; /* global character block with random character */ ms_distr_t *distr; /* distribution from configure file */ char *srv_str; /* string includes servers information */ char *cfg_file; /* configure file name */ ms_mcd_server_t *servers; /* servers array */ uint32_t total_srv_cnt; /* total servers count of the servers array */ uint32_t srv_cnt; /* servers count */ ms_key_distr_t *key_distr; /* array of key distribution */ int total_key_rng_cnt; /* total key range count of the array */ int key_rng_cnt; /* actual key range count */ ms_value_distr_t *value_distr; /* array of value distribution */ int total_val_rng_cnt; /* total value range count of the array */ int val_rng_cnt; /* actual value range count */ ms_cmd_distr_t cmd_distr[CMD_NULL]; /* total we have CMD_NULL commands */ int cmd_used_count; /* supported command count */ size_t fixed_value_size; /* fixed value size */ size_t avg_val_size; /* average value size */ size_t avg_key_size; /* average value size */ double verify_percent; /* percent of data verification */ double exp_ver_per; /* percent of data verification with expire time */ double overwrite_percent; /* percent of overwrite */ int mult_key_num; /* number of keys used by multi-get once */ size_t win_size; /* item window size per connection */ bool udp; /* whether or not use UDP */ int stat_freq; /* statistic frequency second */ bool reconnect; /* whether it reconnect when connection close */ bool verbose; /* whether it outputs detailed information when verification */ bool facebook_test; /* facebook test, TCP set and multi-get with UDP */ uint32_t sock_per_conn; /* number of socks per connection structure */ bool binary_prot_; /* whether it use binary protocol */ int expected_tps; /* expected throughput */ uint32_t rep_write_srv; /* which servers are used to do replication writing */ } ms_setting_st; extern ms_setting_st ms_setting; /* previous part of initialization of setting structure */ void ms_setting_init_pre(void); /* post part of initialization of setting structure */ void ms_setting_init_post(void); /* clean up the global setting structure */ void ms_setting_cleanup(void); #define UNUSED_ARGUMENT(x) (void) x #ifdef __cplusplus } #endif #endif /* end of MS_SETTING_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_sigsegv.c000066400000000000000000000061731440143131000224310ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #include #include #include #include #include #include "ms_memslap.h" #include "ms_setting.h" /* prototypes */ int ms_setup_sigsegv(void); int ms_setup_sigpipe(void); int ms_setup_sigint(void); /* signal seg reaches, this function will run */ static void ms_signal_segv(int signum, siginfo_t *info, void *ptr) { UNUSED_ARGUMENT(signum); UNUSED_ARGUMENT(info); UNUSED_ARGUMENT(ptr); pthread_mutex_lock(&ms_global.quit_mutex); fprintf(stderr, "Segmentation fault occurred.\nStack trace:\n"); #if 0 pandora_print_callstack(stderr); #endif fprintf(stderr, "End of stack trace\n"); pthread_mutex_unlock(&ms_global.quit_mutex); abort(); } /* signal int reaches, this function will run */ static void ms_signal_int(int signum, siginfo_t *info, void *ptr) { UNUSED_ARGUMENT(signum); UNUSED_ARGUMENT(info); UNUSED_ARGUMENT(ptr); pthread_mutex_lock(&ms_global.quit_mutex); fprintf(stderr, "SIGINT handled.\n"); pthread_mutex_unlock(&ms_global.quit_mutex); exit(1); } /* ms_signal_int */ /** * redirect signal seg * * @return if success, return EXIT_SUCCESS, else return -1 */ int ms_setup_sigsegv(void) { struct sigaction action; memset(&action, 0, sizeof(action)); action.sa_sigaction = ms_signal_segv; action.sa_flags = SA_SIGINFO; if (sigaction(SIGSEGV, &action, NULL) < 0) { perror("sigaction"); return EXIT_SUCCESS; } return -1; } /* ms_setup_sigsegv */ /** * redirect signal pipe * * @return if success, return EXIT_SUCCESS, else return -1 */ int ms_setup_sigpipe(void) { /* ignore the SIGPIPE signal */ signal(SIGPIPE, SIG_IGN); return -1; } /* ms_setup_sigpipe */ /** * redirect signal int * * @return if success, return EXIT_SUCCESS, else return -1 */ int ms_setup_sigint(void) { struct sigaction action_3; memset(&action_3, 0, sizeof(action_3)); action_3.sa_sigaction = ms_signal_int; action_3.sa_flags = SA_SIGINFO; if (sigaction(SIGINT, &action_3, NULL) < 0) { perror("sigaction"); return EXIT_SUCCESS; } return -1; } /* ms_setup_sigint */ #ifndef SIGSEGV_NO_AUTO_INIT static void __attribute((constructor)) ms_init(void) { ms_setup_sigsegv(); ms_setup_sigpipe(); ms_setup_sigint(); } #endif libmemcached-1.1.4/contrib/bin/memaslap/ms_sigsegv.h000066400000000000000000000023011440143131000224230ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_SIGSEGV_H #define MS_SIGSEGV_H #ifdef __cplusplus extern "C" { #endif /* redirect signal seg */ int ms_setup_sigsegv(void); /* redirect signal pipe */ int ms_setup_sigpipe(void); /* redirect signal int */ int ms_setup_sigint(void); #ifdef __cplusplus } #endif #endif /* end of MS_SIGSEGV_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_stats.c000066400000000000000000000171061440143131000221160ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #include #include "ms_stats.h" #define array_size(x) (sizeof(x) / sizeof((x)[0])) static int ms_local_log2(uint64_t value); static uint64_t ms_get_events(ms_stat_t *stat); /** * get the index of local log2 array * * @param value * * @return return the index of local log2 array */ static int ms_local_log2(uint64_t value) { int result = 0; while (result <= 63 && ((uint64_t) 1 << result) < value) { result++; } return result; } /* ms_local_log2 */ /** * initialize statistic structure * * @param stat, pointer of the statistic structure * @param name, name of the statistic */ void ms_init_stats(ms_stat_t *stat, const char *name) { memset(stat, 0, sizeof(*stat)); stat->name = (char *) name; stat->min_time = (uint64_t) -1; stat->max_time = 0; stat->period_min_time = (uint64_t) -1; stat->period_max_time = 0; stat->log_product = 0; stat->total_time = 0; stat->pre_total_time = 0; stat->squares = 0; stat->pre_squares = 0; stat->pre_events = 0; stat->pre_log_product = 0; stat->get_miss = 0; stat->pre_get_miss = 0; } /* ms_init_stats */ /** * record one event * * @param stat, pointer of the statistic structure * @param total_time, response time of the command * @param get_miss, whether it gets miss */ void ms_record_event(ms_stat_t *stat, uint64_t total_time, int get_miss) { stat->total_time += total_time; if (total_time < stat->min_time) { stat->min_time = total_time; } if (total_time > stat->max_time) { stat->max_time = total_time; } if (total_time < stat->period_min_time) { stat->period_min_time = total_time; } if (total_time > stat->period_max_time) { stat->period_max_time = total_time; } if (get_miss) { stat->get_miss++; } stat->dist[ms_local_log2(total_time)]++; stat->squares += (double) (total_time * total_time); if (total_time) { stat->log_product += log((double) total_time); } } /* ms_record_event */ /** * get the events count * * @param stat, pointer of the statistic structure * * @return total events recorded */ #if HAVE_TSAN __attribute__ (( no_sanitize_thread, no_sanitize("thread"))) #endif static uint64_t ms_get_events(ms_stat_t *stat) { uint64_t events = 0; for (uint32_t i = 0; i < array_size(stat->dist); i++) { events += stat->dist[i]; } return events; } /* ms_get_events */ /** * dump the statistics * * @param stat, pointer of the statistic structure */ void ms_dump_stats(ms_stat_t *stat) { uint64_t events = 0; int max_non_zero = 0; int min_non_zero = 0; double average = 0; for (uint32_t i = 0; i < array_size(stat->dist); i++) { events += stat->dist[i]; if (stat->dist[i]) { max_non_zero = (int) i; } } if (events == 0) { return; } average = (double) (stat->total_time / events); printf("%s Statistics (%lld events)\n", stat->name, (long long) events); printf(" Min: %8lld\n", (long long) stat->min_time); printf(" Max: %8lld\n", (long long) stat->max_time); printf(" Avg: %8lld\n", (long long) (stat->total_time / events)); printf(" Geo: %8.2lf\n", exp(stat->log_product / (double) events)); if (events > 1) { printf(" Std: %8.2lf\n", sqrt((stat->squares - (double) events * average * average) / ((double) events - 1))); } printf(" Log2 Dist:"); for (int i = 0; i <= max_non_zero - 4; i += 4) { if ((stat->dist[i + 0]) || (stat->dist[i + 1]) || (stat->dist[i + 2]) || (stat->dist[i + 3])) { min_non_zero = i; break; } } for (int i = min_non_zero; i <= max_non_zero; i++) { if ((i % 4) == 0) { printf("\n %2d:", (int) i); } printf(" %6" PRIu64, stat->dist[i]); } printf("\n\n"); } /* ms_dump_stats */ /** * dump the format statistics * * @param stat, pointer of the statistic structure * @param run_time, the total run time * @param freq, statistic frequency * @param obj_size, average object size */ #if HAVE_TSAN __attribute__ (( no_sanitize_thread, no_sanitize("thread"))) #endif void ms_dump_format_stats(ms_stat_t *stat, int run_time, int freq, int obj_size) { uint64_t events = 0; double global_average = 0; uint64_t global_tps = 0; double global_rate = 0; double global_std = 0; double global_log = 0; double period_average = 0; uint64_t period_tps = 0; double period_rate = 0; double period_std = 0; double period_log = 0; if ((events = ms_get_events(stat)) == 0) { return; } global_average = (double) (stat->total_time / events); global_tps = events / (uint64_t) run_time; global_rate = (double) events * obj_size / 1024 / 1024 / run_time; global_std = sqrt((stat->squares - (double) events * global_average * global_average) / (double) (events - 1)); global_log = exp(stat->log_product / (double) events); uint64_t diff_time = stat->total_time - stat->pre_total_time; uint64_t diff_events = events - stat->pre_events; if (diff_events >= 1) { period_average = (double) (diff_time / diff_events); period_tps = diff_events / (uint64_t) freq; period_rate = (double) diff_events * obj_size / 1024 / 1024 / freq; double diff_squares = (double) stat->squares - (double) stat->pre_squares; period_std = sqrt((diff_squares - (double) diff_events * period_average * period_average) / (double) (diff_events - 1)); double diff_log_product = stat->log_product - stat->pre_log_product; period_log = exp(diff_log_product / (double) diff_events); } printf("%s Statistics\n", stat->name); printf("%-8s %-8s %-12s %-12s %-10s %-10s %-8s %-10s %-10s %-10s %-10s\n", "Type", "Time(s)", "Ops", "TPS(ops/s)", "Net(M/s)", "Get_miss", "Min(us)", "Max(us)", "Avg(us)", "Std_dev", "Geo_dist"); printf("%-8s %-8d %-12llu %-12lld %-10.1f %-10lld %-8lld %-10lld %-10lld %-10.2f %.2f\n", "Period", freq, (long long) diff_events, (long long) period_tps, period_rate, (long long) (stat->get_miss - stat->pre_get_miss), (long long) stat->period_min_time, (long long) stat->period_max_time, (long long) period_average, period_std, period_log); printf("%-8s %-8d %-12llu %-12lld %-10.1f %-10lld %-8lld %-10lld %-10lld %-10.2f %.2f\n\n", "Global", run_time, (long long) events, (long long) global_tps, global_rate, (long long) stat->get_miss, (long long) stat->min_time, (long long) stat->max_time, (long long) global_average, global_std, global_log); stat->pre_events = events; stat->pre_squares = (uint64_t) stat->squares; stat->pre_total_time = stat->total_time; stat->pre_log_product = stat->log_product; stat->period_min_time = (uint64_t) -1; stat->period_max_time = 0; stat->pre_get_miss = stat->get_miss; } /* ms_dump_format_stats */ libmemcached-1.1.4/contrib/bin/memaslap/ms_stats.h000066400000000000000000000036061440143131000221230ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_STAT_H #define MS_STAT_H #include #include #include #include #include #include #ifdef __cplusplus extern "C" { #endif /* statistic structure of response time */ typedef struct { char *name; uint64_t total_time; uint64_t min_time; uint64_t max_time; uint64_t get_miss; uint64_t dist[65]; double squares; double log_product; uint64_t period_min_time; uint64_t period_max_time; uint64_t pre_get_miss; uint64_t pre_events; uint64_t pre_total_time; uint64_t pre_squares; double pre_log_product; } ms_stat_t; /* initialize statistic */ void ms_init_stats(ms_stat_t *stat, const char *name); /* record one event */ void ms_record_event(ms_stat_t *stat, uint64_t time, int get_miss); /* dump the statistics */ void ms_dump_stats(ms_stat_t *stat); /* dump the format statistics */ void ms_dump_format_stats(ms_stat_t *stat, int run_time, int freq, int obj_size); #ifdef __cplusplus } #endif #endif /* MS_STAT_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_task.c000066400000000000000000000660371440143131000217310ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #if defined(HAVE_SYS_TIME_H) # include #endif #include #include #include "ms_thread.h" #include "ms_setting.h" #include "ms_atomic.h" /* command distribution adjustment cycle */ #define CMD_DISTR_ADJUST_CYCLE 1000 #define DISADJUST_FACTOR \ 0.03 /** \ * In one adjustment cycle, if undo set or get \ * operations proportion is more than 3% , means \ * there are too many new item or need more new \ * item in the window. This factor shows it. \ */ /* get item from task window */ static ms_task_item_t *ms_get_cur_opt_item(ms_conn_t *c); static ms_task_item_t *ms_get_next_get_item(ms_conn_t *c); static ms_task_item_t *ms_get_next_set_item(ms_conn_t *c); static ms_task_item_t *ms_get_random_overwrite_item(ms_conn_t *c); /* select next operation to do */ static void ms_select_opt(ms_conn_t *c, ms_task_t *task); /* set and get speed estimate for controlling and adjustment */ static bool ms_is_set_too_fast(ms_task_t *task); static bool ms_is_get_too_fast(ms_task_t *task); static void ms_kick_out_item(ms_task_item_t *item); /* miss rate adjustment */ static bool ms_need_overwrite_item(ms_task_t *task); static bool ms_adjust_opt(ms_conn_t *c, ms_task_t *task); /* deal with data verification initialization */ static void ms_task_data_verify_init(ms_task_t *task); static void ms_task_expire_verify_init(ms_task_t *task); /* select a new task to do */ static ms_task_t *ms_get_task(ms_conn_t *c, bool warmup); /* run the selected task */ static void ms_update_set_result(ms_conn_t *c, ms_task_item_t *item); static void ms_update_stat_result(ms_conn_t *c); static void ms_update_multi_get_result(ms_conn_t *c); static void ms_update_single_get_result(ms_conn_t *c, ms_task_item_t *item); static void ms_update_task_result(ms_conn_t *c); static void ms_single_getset_task_sch(ms_conn_t *c); static void ms_multi_getset_task_sch(ms_conn_t *c); static void ms_send_signal(ms_sync_lock_t *sync_lock); static void ms_warmup_server(ms_conn_t *c); static int ms_run_getset_task(ms_conn_t *c); /** * used to get the current operation item(object) * * @param c, pointer of the concurrency * * @return ms_task_item_t*, current operating item */ static ms_task_item_t *ms_get_cur_opt_item(ms_conn_t *c) { return c->curr_task.item; } /** * used to get the next item to do get operation * * @param c, pointer of the concurrency * * @return ms_task_item_t*, the pointer of the next item to do * get operation */ static ms_task_item_t *ms_get_next_get_item(ms_conn_t *c) { ms_task_item_t *item = NULL; if (c->set_cursor <= 0) { /* the first item in the window */ item = &c->item_win[0]; } else if (c->set_cursor > 0 && c->set_cursor < (uint32_t) c->win_size) { /* random get one item set before */ item = &c->item_win[random() % (int64_t) c->set_cursor]; } else { /* random get one item from the window */ item = &c->item_win[random() % c->win_size]; } return item; } /* ms_get_next_get_item */ /** * used to get the next item to do set operation * * @param c, pointer of the concurrency * * @return ms_task_item_t*, the pointer of the next item to do * set operation */ static ms_task_item_t *ms_get_next_set_item(ms_conn_t *c) { /** * when a set command successes, the cursor will plus 1. If set * fails, the cursor doesn't change. it isn't necessary to * increase the cursor here. */ return &c->item_win[(int64_t) c->set_cursor % c->win_size]; } /** * If we need do overwrite, we could select a item set before. * This function is used to get a item set before to do * overwrite. * * @param c, pointer of the concurrency * * @return ms_task_item_t*, the pointer of the previous item of * set operation */ static ms_task_item_t *ms_get_random_overwrite_item(ms_conn_t *c) { return ms_get_next_get_item(c); } /* ms_get_random_overwrite_item */ /** * According to the proportion of operations(get or set), select * an operation to do. * * @param c, pointer of the concurrency * @param task, pointer of current task in the concurrency */ static void ms_select_opt(ms_conn_t *c, ms_task_t *task) { double get_prop = ms_setting.cmd_distr[CMD_GET].cmd_prop; double set_prop = ms_setting.cmd_distr[CMD_SET].cmd_prop; /* update cycle operation number if necessary */ if ((task->cycle_undo_get == 0) || (task->cycle_undo_set == 0)) { task->cycle_undo_get += (int) (CMD_DISTR_ADJUST_CYCLE * get_prop); task->cycle_undo_set += (int) (CMD_DISTR_ADJUST_CYCLE * set_prop); } /** * According to operation distribution to choose doing which * operation. If it can't set new object to sever, just change * to do get operation. */ if ((set_prop > PROP_ERROR) && ((double) task->get_opt * set_prop >= (double) task->set_opt * get_prop)) { task->cmd = CMD_SET; task->item = ms_get_next_set_item(c); } else { task->cmd = CMD_GET; task->item = ms_get_next_get_item(c); } } /* ms_select_opt */ /** * used to judge whether the number of get operations done is * more than expected number of get operations to do right now. * * @param task, pointer of current task in the concurrency * * @return bool, if get too fast, return true, else return false */ static bool ms_is_get_too_fast(ms_task_t *task) { double get_prop = ms_setting.cmd_distr[CMD_GET].cmd_prop; double set_prop = ms_setting.cmd_distr[CMD_SET].cmd_prop; /* no get operation */ if (get_prop < PROP_ERROR) { return false; } int max_undo_set = (int) (set_prop / get_prop * (1.0 + DISADJUST_FACTOR)) * task->cycle_undo_get; if (((double) task->get_opt * set_prop > (double) task->set_opt * get_prop) && (task->cycle_undo_set > max_undo_set)) { return true; } return false; } /* ms_is_get_too_fast */ /** * used to judge whether the number of set operations done is * more than expected number of set operations to do right now. * * @param task, pointer of current task in the concurrency * * @return bool, if set too fast, return true, else return false */ static bool ms_is_set_too_fast(ms_task_t *task) { double get_prop = ms_setting.cmd_distr[CMD_GET].cmd_prop; double set_prop = ms_setting.cmd_distr[CMD_SET].cmd_prop; /* no set operation */ if (set_prop < PROP_ERROR) { return false; } /* If it does set operation too fast, skip some */ int max_undo_get = (int) ((get_prop / set_prop * (1.0 + DISADJUST_FACTOR)) * (double) task->cycle_undo_set); if (((double) task->get_opt * set_prop < (double) task->set_opt * get_prop) && (task->cycle_undo_get > max_undo_get)) { return true; } return false; } /* ms_is_set_too_fast */ /** * kick out the old item in the window, and add a new item to * overwrite the old item. When we don't want to do overwrite * object, and the current item to do set operation is an old * item, we could kick out the old item and add a new item. Then * we can ensure we set new object every time. * * @param item, pointer of task item which includes the object * information */ static void ms_kick_out_item(ms_task_item_t *item) { /* allocate a new item */ item->key_prefix = ms_get_key_prefix(); item->key_suffix_offset++; item->value_offset = INVALID_OFFSET; /* new item use invalid value offset */ item->client_time = 0; } /* ms_kick_out_item */ /** * used to judge whether we need overwrite object based on the * options user specified * * @param task, pointer of current task in the concurrency * * @return bool, if need overwrite, return true, else return * false */ static bool ms_need_overwrite_item(ms_task_t *task) { ms_task_item_t *item = task->item; assert(item); assert(task->cmd == CMD_SET); /** * according to data overwrite percent to determine if do data * overwrite. */ if (task->overwrite_set < (double) task->set_opt * ms_setting.overwrite_percent) { return true; } return false; } /* ms_need_overwirte_item */ /** * used to adjust operation. the function must be called after * select operation. the function change get operation to set * operation, or set operation to get operation based on the * current case. * * @param c, pointer of the concurrency * @param task, pointer of current task in the concurrency * * @return bool, if success, return true, else return false */ static bool ms_adjust_opt(ms_conn_t *c, ms_task_t *task) { ms_task_item_t *item = task->item; assert(item); if (task->cmd == CMD_SET) { /* If did set operation too fast, skip some */ if (ms_is_set_too_fast(task)) { /* get the item instead */ if (item->value_offset != INVALID_OFFSET) { task->cmd = CMD_GET; return true; } } /* If the current item is not a new item, kick it out */ if (item->value_offset != INVALID_OFFSET) { if (ms_need_overwrite_item(task)) { /* overwrite */ task->overwrite_set++; } else { /* kick out the current item to do set operation */ ms_kick_out_item(item); } } else /* it's a new item */ { /* need overwrite */ if (ms_need_overwrite_item(task)) { /** * overwrite not use the item with current set cursor, revert * set cursor. */ c->set_cursor--; item = ms_get_random_overwrite_item(c); if (item->value_offset != INVALID_OFFSET) { task->item = item; task->overwrite_set++; } else /* item is a new item */ { /* select the item to run, and cancel overwrite */ task->item = item; } } } task->cmd = CMD_SET; return true; } else { if (item->value_offset == INVALID_OFFSET) { task->cmd = CMD_SET; return true; } /** * If It does get operation too fast, it will change the * operation to set. */ if (ms_is_get_too_fast(task)) { /* don't kick out the first item in the window */ if (!ms_is_set_too_fast(task)) { ms_kick_out_item(item); task->cmd = CMD_SET; return true; } else { return false; } } assert(item->value_offset != INVALID_OFFSET); task->cmd = CMD_GET; return true; } } /* ms_adjust_opt */ /** * used to initialize the task which need verify data. * * @param task, pointer of current task in the concurrency */ static void ms_task_data_verify_init(ms_task_t *task) { ms_task_item_t *item = task->item; assert(item); assert(task->cmd == CMD_GET); /** * according to data verification percent to determine if do * data verification. */ if (task->verified_get < (double) task->get_opt * ms_setting.verify_percent) { /** * currently it doesn't do verify, just increase the counter, * and do verification next proper get command */ if ((task->item->value_offset != INVALID_OFFSET) && (item->exp_time == 0)) { task->verify = true; task->finish_verify = false; task->verified_get++; } } } /* ms_task_data_verify_init */ /** * used to initialize the task which need verify expire time. * * @param task, pointer of current task in the concurrency */ static void ms_task_expire_verify_init(ms_task_t *task) { ms_task_item_t *item = task->item; assert(item); assert(task->cmd == CMD_GET); assert(item->exp_time > 0); task->verify = true; task->finish_verify = false; } /* ms_task_expire_verify_init */ /** * used to get one task, the function initializes the task * structure. * * @param c, pointer of the concurrency * @param warmup, whether it need warmup * * @return ms_task_t*, pointer of current task in the * concurrency */ static ms_task_t *ms_get_task(ms_conn_t *c, bool warmup) { ms_task_t *task = &c->curr_task; while (1) { task->verify = false; task->finish_verify = true; task->get_miss = true; if (warmup) { task->cmd = CMD_SET; task->item = ms_get_next_set_item(c); return task; } /* according to operation distribution to choose doing which operation */ ms_select_opt(c, task); if (!ms_adjust_opt(c, task)) { continue; } if ((ms_setting.verify_percent > 0) && (task->cmd == CMD_GET)) { ms_task_data_verify_init(task); } if ((ms_setting.exp_ver_per > 0) && (task->cmd == CMD_GET) && (task->item->exp_time > 0)) { ms_task_expire_verify_init(task); } break; } /** * Only update get and delete counter, set counter will be * updated after set operation successes. */ if (task->cmd == CMD_GET) { task->get_opt++; task->cycle_undo_get--; } return task; } /* ms_get_task */ /** * send a signal to the main monitor thread * * @param sync_lock, pointer of the lock */ static void ms_send_signal(ms_sync_lock_t *sync_lock) { pthread_mutex_lock(&sync_lock->lock); sync_lock->count++; pthread_cond_signal(&sync_lock->cond); pthread_mutex_unlock(&sync_lock->lock); } /* ms_send_signal */ /** * If user only want to do get operation, but there is no object * in server , so we use this function to warmup the server, and * set some objects to server. It runs at the beginning of task. * * @param c, pointer of the concurrency */ static void ms_warmup_server(ms_conn_t *c) { ms_task_t *task; ms_task_item_t *item; /** * Extra one loop to get the last command returned state. * Normally it gets the previous command returned state. */ if ((c->remain_warmup_num >= 0) && (c->remain_warmup_num != c->warmup_num)) { item = ms_get_cur_opt_item(c); /* only update the set command result state for data verification */ if ((c->precmd.cmd == CMD_SET) && (c->precmd.retstat == MCD_STORED)) { item->value_offset = item->key_suffix_offset; /* set success, update counter */ c->set_cursor++; } else if (c->precmd.cmd == CMD_SET && c->precmd.retstat != MCD_STORED) { printf("key: %" PRIx64 " didn't set success\n", item->key_prefix); } } /* the last time don't run a task */ if (c->remain_warmup_num-- > 0) { /* operate next task item */ task = ms_get_task(c, true); item = task->item; ms_mcd_set(c, item); } /** * finish warming up server, wait all connects initialize * complete. Then all connects can start do task at the same * time. */ if (c->remain_warmup_num == -1) { ms_send_signal(&ms_global.warmup_lock); c->remain_warmup_num--; /* never run the if branch */ } } /* ms_warmup_server */ /** * dispatch single get and set task * * @param c, pointer of the concurrency */ static void ms_single_getset_task_sch(ms_conn_t *c) { ms_task_t *task; ms_task_item_t *item; /* the last time don't run a task */ if (c->remain_exec_num-- > 0) { task = ms_get_task(c, false); item = task->item; if (task->cmd == CMD_SET) { ms_mcd_set(c, item); } else if (task->cmd == CMD_GET) { assert(task->cmd == CMD_GET); ms_mcd_get(c, item); } } } /* ms_single_getset_task_sch */ /** * dispatch multi-get and set task * * @param c, pointer of the concurrency */ static void ms_multi_getset_task_sch(ms_conn_t *c) { ms_task_t *task; ms_mlget_task_item_t *mlget_item; while (1) { if (c->remain_exec_num-- > 0) { task = ms_get_task(c, false); if (task->cmd == CMD_SET) /* just do it */ { ms_mcd_set(c, task->item); break; } else { assert(task->cmd == CMD_GET); mlget_item = &c->mlget_task.mlget_item[c->mlget_task.mlget_num]; mlget_item->item = task->item; mlget_item->verify = task->verify; mlget_item->finish_verify = task->finish_verify; mlget_item->get_miss = task->get_miss; c->mlget_task.mlget_num++; /* enough multi-get task items can be done */ if ((c->mlget_task.mlget_num >= ms_setting.mult_key_num) || ((c->remain_exec_num == 0) && (c->mlget_task.mlget_num > 0))) { ms_mcd_mlget(c); break; } } } else { if ((c->remain_exec_num <= 0) && (c->mlget_task.mlget_num > 0)) { ms_mcd_mlget(c); } break; } } } /* ms_multi_getset_task_sch */ /** * calculate the difference value of two time points * * @param start_time, the start time * @param end_time, the end time * * @return uint64_t, the difference value between start_time and end_time in us */ int64_t ms_time_diff(struct timeval *start_time, struct timeval *end_time) { int64_t endtime = end_time->tv_sec * 1000000 + end_time->tv_usec; int64_t starttime = start_time->tv_sec * 1000000 + start_time->tv_usec; assert(endtime >= starttime); return endtime - starttime; } /* ms_time_diff */ /** * after get the response from server for multi-get, the * function update the state of the task and do data verify if * necessary. * * @param c, pointer of the concurrency */ static void ms_update_multi_get_result(ms_conn_t *c) { ms_mlget_task_item_t *mlget_item; ms_task_item_t *item; char *orignval = NULL; char *orignkey = NULL; if (c == NULL) { return; } assert(c); for (int i = 0; i < c->mlget_task.mlget_num; i++) { mlget_item = &c->mlget_task.mlget_item[i]; item = mlget_item->item; orignval = &ms_setting.char_block[item->value_offset]; orignkey = &ms_setting.char_block[item->key_suffix_offset]; /* update get miss counter */ if (mlget_item->get_miss) { atomic_add_size(&ms_stats.get_misses, 1); } /* get nothing from server for this task item */ if (mlget_item->verify && !mlget_item->finish_verify) { /* verify expire time if necessary */ if (item->exp_time > 0) { struct timeval curr_time; gettimeofday(&curr_time, NULL); /* object doesn't expire but can't get it now */ if (curr_time.tv_sec - item->client_time < item->exp_time - EXPIRE_TIME_ERROR) { atomic_add_size(&ms_stats.unexp_unget, 1); if (ms_setting.verbose) { char set_time[64]; char cur_time[64]; strftime(set_time, 64, "%Y-%m-%d %H:%M:%S", localtime(&item->client_time)); strftime(cur_time, 64, "%Y-%m-%d %H:%M:%S", localtime(&curr_time.tv_sec)); fprintf(stderr, "\n\t<%d expire time verification failed, object " "doesn't expire but can't get it now\n" "\tkey len: %d\n" "\tkey: %" PRIx64 " %.*s\n" "\tset time: %s current time: %s " "diff time: %d expire time: %d\n" "\texpected data len: %d\n" "\texpected data: %.*s\n" "\treceived data: \n", c->sfd, item->key_size, item->key_prefix, item->key_size - (int) KEY_PREFIX_SIZE, orignkey, set_time, cur_time, (int) (curr_time.tv_sec - item->client_time), item->exp_time, item->value_size, item->value_size, orignval); fflush(stderr); } } } else { atomic_add_size(&ms_stats.vef_miss, 1); if (ms_setting.verbose) { fprintf(stderr, "\n<%d data verification failed\n" "\tkey len: %d\n" "\tkey: %" PRIx64 " %.*s\n" "\texpected data len: %d\n" "\texpected data: %.*s\n" "\treceived data: \n", c->sfd, item->key_size, item->key_prefix, item->key_size - (int) KEY_PREFIX_SIZE, orignkey, item->value_size, item->value_size, orignval); fflush(stderr); } } } } c->mlget_task.mlget_num = 0; c->mlget_task.value_index = INVALID_OFFSET; } /* ms_update_multi_get_result */ /** * after get the response from server for single get, the * function update the state of the task and do data verify if * necessary. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information */ static void ms_update_single_get_result(ms_conn_t *c, ms_task_item_t *item) { char *orignval = NULL; char *orignkey = NULL; if ((c == NULL) || (item == NULL)) { return; } assert(c); assert(item); orignval = &ms_setting.char_block[item->value_offset]; orignkey = &ms_setting.char_block[item->key_suffix_offset]; /* update get miss counter */ if ((c->precmd.cmd == CMD_GET) && c->curr_task.get_miss) { atomic_add_size(&ms_stats.get_misses, 1); } /* get nothing from server for this task item */ if ((c->precmd.cmd == CMD_GET) && c->curr_task.verify && !c->curr_task.finish_verify) { /* verify expire time if necessary */ if (item->exp_time > 0) { struct timeval curr_time; gettimeofday(&curr_time, NULL); /* object doesn't expire but can't get it now */ if (curr_time.tv_sec - item->client_time < item->exp_time - EXPIRE_TIME_ERROR) { atomic_add_size(&ms_stats.unexp_unget, 1); if (ms_setting.verbose) { char set_time[64]; char cur_time[64]; strftime(set_time, 64, "%Y-%m-%d %H:%M:%S", localtime(&item->client_time)); strftime(cur_time, 64, "%Y-%m-%d %H:%M:%S", localtime(&curr_time.tv_sec)); fprintf(stderr, "\n\t<%d expire time verification failed, object " "doesn't expire but can't get it now\n" "\tkey len: %d\n" "\tkey: %" PRIx64 " %.*s\n" "\tset time: %s current time: %s " "diff time: %d expire time: %d\n" "\texpected data len: %d\n" "\texpected data: %.*s\n" "\treceived data: \n", c->sfd, item->key_size, item->key_prefix, item->key_size - (int) KEY_PREFIX_SIZE, orignkey, set_time, cur_time, (int) (curr_time.tv_sec - item->client_time), item->exp_time, item->value_size, item->value_size, orignval); fflush(stderr); } } } else { atomic_add_size(&ms_stats.vef_miss, 1); if (ms_setting.verbose) { fprintf(stderr, "\n<%d data verification failed\n" "\tkey len: %d\n" "\tkey: %" PRIx64 " %.*s\n" "\texpected data len: %d\n" "\texpected data: %.*s\n" "\treceived data: \n", c->sfd, item->key_size, item->key_prefix, item->key_size - (int) KEY_PREFIX_SIZE, orignkey, item->value_size, item->value_size, orignval); fflush(stderr); } } } } /* ms_update_single_get_result */ /** * after get the response from server for set the function * update the state of the task and do data verify if necessary. * * @param c, pointer of the concurrency * @param item, pointer of task item which includes the object * information */ static void ms_update_set_result(ms_conn_t *c, ms_task_item_t *item) { if ((c == NULL) || (item == NULL)) { return; } assert(c); assert(item); if (c->precmd.cmd == CMD_SET) { switch (c->precmd.retstat) { case MCD_STORED: if (item->value_offset == INVALID_OFFSET) { /* first set with the same offset of key suffix */ item->value_offset = item->key_suffix_offset; } else { /* not first set, just increase the value offset */ item->value_offset += 1; } /* set successes, update counter */ c->set_cursor++; c->curr_task.set_opt++; c->curr_task.cycle_undo_set--; break; case MCD_SERVER_ERROR: default: break; } /* switch */ } } /* ms_update_set_result */ /** * update the response time result * * @param c, pointer of the concurrency */ static void ms_update_stat_result(ms_conn_t *c) { bool get_miss = false; if (c == NULL) { return; } assert(c); gettimeofday(&c->end_time, NULL); uint64_t time_diff = (uint64_t) ms_time_diff(&c->start_time, &c->end_time); pthread_mutex_lock(&ms_statistic.stat_mutex); switch (c->precmd.cmd) { case CMD_SET: ms_record_event(&ms_statistic.set_stat, time_diff, false); break; case CMD_GET: if (c->curr_task.get_miss) { get_miss = true; } ms_record_event(&ms_statistic.get_stat, time_diff, get_miss); break; default: break; } /* switch */ ms_record_event(&ms_statistic.total_stat, time_diff, get_miss); pthread_mutex_unlock(&ms_statistic.stat_mutex); } /* ms_update_stat_result */ /** * after get response from server for the current operation, and * before doing the next operation, update the state of the * current operation. * * @param c, pointer of the concurrency */ static void ms_update_task_result(ms_conn_t *c) { ms_task_item_t *item; if (c == NULL) { return; } assert(c); item = ms_get_cur_opt_item(c); if (item == NULL) { return; } assert(item); ms_update_set_result(c, item); if ((ms_setting.stat_freq > 0) && ((c->precmd.cmd == CMD_SET) || (c->precmd.cmd == CMD_GET))) { ms_update_stat_result(c); } /* update multi-get task item */ if (((ms_setting.mult_key_num > 1) && (c->mlget_task.mlget_num >= ms_setting.mult_key_num)) || ((c->remain_exec_num == 0) && (c->mlget_task.mlget_num > 0))) { ms_update_multi_get_result(c); } else { ms_update_single_get_result(c, item); } } /* ms_update_task_result */ /** * run get and set operation * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ static int ms_run_getset_task(ms_conn_t *c) { /** * extra one loop to get the last command return state. get the * last command return state. */ if ((c->remain_exec_num >= 0) && (c->remain_exec_num != c->exec_num)) { ms_update_task_result(c); } /* multi-get */ if (ms_setting.mult_key_num > 1) { /* operate next task item */ ms_multi_getset_task_sch(c); } else { /* operate next task item */ ms_single_getset_task_sch(c); } /* no task to do, exit */ if ((c->remain_exec_num == -1) || ms_global.time_out) { return -1; } return EXIT_SUCCESS; } /* ms_run_getset_task */ /** * the state machine call the function to execute task. * * @param c, pointer of the concurrency * * @return int, if success, return EXIT_SUCCESS, else return -1 */ int ms_exec_task(struct conn *c) { if (!ms_global.finish_warmup) { ms_warmup_server(c); } else { if (ms_run_getset_task(c)) { return -1; } } return EXIT_SUCCESS; } /* ms_exec_task */ libmemcached-1.1.4/contrib/bin/memaslap/ms_task.h000066400000000000000000000072561440143131000217340ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_TASK_H #define MS_TASK_H #include #include #if !defined(__cplusplus) # include #endif #include #ifdef __cplusplus extern "C" { #endif #define UNIT_ITEMS_COUNT 1024 /* each window unit has 1024 items */ #define KEY_PREFIX_SIZE (sizeof(uint64_t)) /* key prefix length: 8 bytes */ #define INVALID_OFFSET (-1) /* invalid offset in the character table */ #define FIXED_EXPIRE_TIME 60 /* default expire time is 60s */ #define EXPIRE_TIME_ERROR 5 /* default expire time error is 5s */ /* information of a task item(object) */ typedef struct task_item { uint64_t key_prefix; /* prefix of the key, 8 bytes, binary */ int key_size; /* key size */ int key_suffix_offset; /* suffix offset in the global character table */ int value_size; /* data size */ int value_offset; /* data offset in the global character table */ time_t client_time; /* the current client time */ int exp_time; /* expire time */ } ms_task_item_t; /* task item for multi-get */ typedef struct mlget_task_item { ms_task_item_t *item; /* task item */ bool verify; /* whether verify data or not */ bool finish_verify; /* whether finish data verify or not */ bool get_miss; /* whether get miss or not */ } ms_mlget_task_item_t; /* information of multi-get task */ typedef struct mlget_task { ms_mlget_task_item_t *mlget_item; /* multi-get task array */ int mlget_num; /* how many tasks in mlget_task array */ int value_index; /* the nth value received by the connect, for multi-get */ } ms_mlget_task_t; /* structure used to store the state of the running task */ typedef struct task { int cmd; /* command name */ bool verify; /* whether verify data or not */ bool finish_verify; /* whether finish data verify or not */ bool get_miss; /* whether get miss or not */ ms_task_item_t *item; /* task item */ /* counter for command distribution adjustment */ uint64_t get_opt; /* number of total get operations */ uint64_t set_opt; /* number of total set operations, no including warmup set count */ int cycle_undo_get; /* number of undo get in an adjustment cycle */ int cycle_undo_set; /* number of undo set in an adjustment cycle */ uint64_t verified_get; /* number of total verified get operations */ uint64_t overwrite_set; /* number of total overwrite set operations */ } ms_task_t; struct conn; /* the state machine call the function to execute task.*/ int ms_exec_task(struct conn *c); /* calculate the difference value of two time points */ int64_t ms_time_diff(struct timeval *start_time, struct timeval *end_time); #ifdef __cplusplus } #endif #endif /* end of MS_TASK_H */ libmemcached-1.1.4/contrib/bin/memaslap/ms_thread.c000066400000000000000000000214171440143131000222270ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #include "mem_config.h" #if defined(HAVE_SYS_TIME_H) # include #endif #include #include "ms_thread.h" #include "ms_setting.h" #include "ms_atomic.h" /* global variable */ pthread_key_t ms_thread_key; /* array of thread context structure, each thread has a thread context structure */ static ms_thread_ctx_t *ms_thread_ctx; /* functions */ static void ms_set_current_time(void); static void ms_check_sock_timeout(void); static void ms_clock_handler(const int fd, const short which, void *arg); static uint32_t ms_set_thread_cpu_affinity(uint32_t cpu); static int ms_setup_thread(ms_thread_ctx_t *thread_ctx); static void *ms_worker_libevent(void *arg); static void ms_create_worker(void *(*func)(void *), ms_thread_ctx_t *arg); /** * time-sensitive callers can call it by hand with this, * outside the normal ever-1-second timer */ static void ms_set_current_time() { struct timeval timer; ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); gettimeofday(&timer, NULL); ms_thread->curr_time = (rel_time_t) timer.tv_sec; } /* ms_set_current_time */ /** * used to check whether UDP of command are waiting timeout * by the ever-1-second timer */ static void ms_check_sock_timeout(void) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); ms_conn_t *c = NULL; int time_diff = 0; for (uint32_t i = 0; i < ms_thread->thread_ctx->nconns; i++) { c = &ms_thread->conn[i]; if (c->udp) { time_diff = (int) (ms_thread->curr_time - (rel_time_t) c->start_time.tv_sec); /* wait time out */ if (time_diff > SOCK_WAIT_TIMEOUT) { /* calculate dropped packets count */ if (c->recvpkt > 0) { atomic_add_size(&ms_stats.pkt_drop, c->packets - c->recvpkt); } atomic_add_size(&ms_stats.udp_timeout, 1); ms_reset_conn(c, true); } } } } /* ms_check_sock_timeout */ /* if disconnect, the ever-1-second timer will call this function to reconnect */ static void ms_reconn_thread_socks(void) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); for (uint32_t i = 0; i < ms_thread->thread_ctx->nconns; i++) { ms_reconn_socks(&ms_thread->conn[i]); } } /* ms_reconn_thread_socks */ /** * the handler of the ever-1-second timer * * @param fd, the descriptors of the socket * @param which, event flags * @param arg, argument */ static void ms_clock_handler(const int fd, const short which, void *arg) { ms_thread_t *ms_thread = pthread_getspecific(ms_thread_key); struct timeval t = {.tv_sec = 1, .tv_usec = 0}; UNUSED_ARGUMENT(fd); UNUSED_ARGUMENT(which); UNUSED_ARGUMENT(arg); ms_set_current_time(); if (ms_thread->initialized) { /* only delete the event if it's actually there. */ evtimer_del(&ms_thread->clock_event); ms_check_sock_timeout(); } else { ms_thread->initialized = true; } ms_reconn_thread_socks(); evtimer_set(&ms_thread->clock_event, ms_clock_handler, 0); event_base_set(ms_thread->base, &ms_thread->clock_event); evtimer_add(&ms_thread->clock_event, &t); } /* ms_clock_handler */ /** * used to bind thread to CPU if the system supports * * @param cpu, cpu index * * @return if success, return EXIT_SUCCESS, else return -1 */ static uint32_t ms_set_thread_cpu_affinity(uint32_t cpu) { uint32_t ret = 0; #ifdef HAVE_CPU_SET_T cpu_set_t cpu_set; CPU_ZERO(&cpu_set); CPU_SET(cpu, &cpu_set); if (sched_setaffinity(0, sizeof(cpu_set_t), &cpu_set) == -1) { fprintf(stderr, "WARNING: Could not set CPU Affinity, continuing...\n"); ret = 1; } #else UNUSED_ARGUMENT(cpu); #endif return ret; } /* ms_set_thread_cpu_affinity */ /** * Set up a thread's information. * * @param thread_ctx, pointer of the thread context structure * * @return if success, return EXIT_SUCCESS, else return -1 */ static int ms_setup_thread(ms_thread_ctx_t *thread_ctx) { ms_thread_t *ms_thread = (ms_thread_t *) calloc(sizeof(*ms_thread), 1); pthread_setspecific(ms_thread_key, (void *) ms_thread); ms_thread->thread_ctx = thread_ctx; ms_thread->nactive_conn = thread_ctx->nconns; ms_thread->initialized = false; static ATOMIC uint32_t cnt = 0; gettimeofday(&ms_thread->startup_time, NULL); ms_thread->base = event_base_new(); if (ms_thread->base == NULL) { if (atomic_add_32_nv(&cnt, 1) == 0) { fprintf(stderr, "Can't allocate event base.\n"); } return -1; } ms_thread->conn = (ms_conn_t *) malloc((size_t) thread_ctx->nconns * sizeof(ms_conn_t)); if (ms_thread->conn == NULL) { if (atomic_add_32_nv(&cnt, 1) == 0) { fprintf(stderr, "Can't allocate concurrency structure for thread descriptors."); } return -1; } memset(ms_thread->conn, 0, (size_t) thread_ctx->nconns * sizeof(ms_conn_t)); for (uint32_t i = 0; i < thread_ctx->nconns; i++) { ms_thread->conn[i].conn_idx = i; if (ms_setup_conn(&ms_thread->conn[i])) { /* only output this error once */ if (atomic_add_32_nv(&cnt, 1) == 0) { fprintf(stderr, "Initializing connection failed.\n"); } return -1; } } return EXIT_SUCCESS; } /* ms_setup_thread */ /** * Worker thread: main event loop * * @param arg, the pointer of argument * * @return void* */ static void *ms_worker_libevent(void *arg) { ms_thread_t *ms_thread = NULL; ms_thread_ctx_t *thread_ctx = (ms_thread_ctx_t *) arg; /** * If system has more than one cpu and supports set cpu * affinity, try to bind each thread to a cpu core; */ if (ms_setting.ncpu > 1) { ms_set_thread_cpu_affinity(thread_ctx->thd_idx % ms_setting.ncpu); } if (ms_setup_thread(thread_ctx)) { exit(1); } /* each thread with a timer */ ms_clock_handler(0, 0, 0); pthread_mutex_lock(&ms_global.init_lock.lock); ms_global.init_lock.count++; pthread_cond_signal(&ms_global.init_lock.cond); pthread_mutex_unlock(&ms_global.init_lock.lock); ms_thread = pthread_getspecific(ms_thread_key); event_base_loop(ms_thread->base, 0); event_base_free(ms_thread->base); free(ms_thread); return NULL; } /* ms_worker_libevent */ /** * Creates a worker thread. * * @param func, the callback function * @param arg, the argument to pass to the callback function */ static void ms_create_worker(void *(*func)(void *), ms_thread_ctx_t *arg) { pthread_attr_t attr; int ret; pthread_attr_init(&attr); if ((ret = pthread_create(&arg->pth_id, &attr, func, arg))) { fprintf(stderr, "Can't create thread: %s.\n", strerror(ret)); exit(1); } } /* ms_create_worker */ /* initialize threads */ void ms_thread_init() { ms_thread_ctx = (ms_thread_ctx_t *) malloc(sizeof(ms_thread_ctx_t) * (size_t) ms_setting.nthreads); if (ms_thread_ctx == NULL) { fprintf(stderr, "Can't allocate thread descriptors."); exit(1); } for (uint32_t i = 0; i < ms_setting.nthreads; i++) { ms_thread_ctx[i].thd_idx = i; ms_thread_ctx[i].nconns = ms_setting.nconns / ms_setting.nthreads; /** * If only one server, all the connections in all threads * connects the same server. For support multi-servers, simple * distribute thread to server. */ ms_thread_ctx[i].srv_idx = i % ms_setting.srv_cnt; ms_thread_ctx[i].tps_perconn = ms_setting.expected_tps / (int) ms_setting.nconns; ms_thread_ctx[i].exec_num_perconn = ms_setting.exec_num / ms_setting.nconns; } if (pthread_key_create(&ms_thread_key, NULL)) { fprintf(stderr, "Can't create pthread keys. Major malfunction!\n"); exit(1); } /* Create threads after we've done all the epoll setup. */ for (uint32_t i = 0; i < ms_setting.nthreads; i++) { ms_create_worker(ms_worker_libevent, &ms_thread_ctx[i]); } } /* ms_thread_init */ /* cleanup some resource of threads when all the threads exit */ void ms_thread_cleanup() { for (uint32_t i = 0; i < ms_setting.nthreads; i++) { pthread_join(ms_thread_ctx[i].pth_id, NULL); } if (ms_thread_ctx) { free(ms_thread_ctx); } pthread_key_delete(ms_thread_key); } /* ms_thread_cleanup */ libmemcached-1.1.4/contrib/bin/memaslap/ms_thread.h000066400000000000000000000046071440143131000222360ustar00rootroot00000000000000/* +--------------------------------------------------------------------+ | libmemcached-awesome - C/C++ Client Library for memcached | +--------------------------------------------------------------------+ | Redistribution and use in source and binary forms, with or without | | modification, are permitted under the terms of the BSD license. | | You should have received a copy of the license in a bundled file | | named LICENSE; in case you did not receive a copy you can review | | the terms online at: https://opensource.org/licenses/BSD-3-Clause | +--------------------------------------------------------------------+ | Copyright (c) 2006-2014 Brian Aker https://datadifferential.com/ | | Copyright (c) 2020 Michael Wallner | +--------------------------------------------------------------------+ */ #ifndef MS_THREAD_H #define MS_THREAD_H #include #include #include "ms_conn.h" #ifdef __cplusplus extern "C" { #endif /** Time relative to server start. Smaller than time_t on 64-bit systems. */ typedef unsigned int rel_time_t; /* Used to store the context of each thread */ typedef struct thread_ctx { uint32_t thd_idx; /* the thread index */ uint32_t nconns; /* how many connections included by the thread */ uint32_t srv_idx; /* index of the thread */ int tps_perconn; /* expected throughput per connection */ int64_t exec_num_perconn; /* execute number per connection */ pthread_t pth_id; } ms_thread_ctx_t; /* Used to store the private variables of each thread */ typedef struct thread { ms_conn_t *conn; /* conn array to store all the conn in the thread */ uint32_t nactive_conn; /* how many connects are active */ ms_thread_ctx_t *thread_ctx; /* thread context from the caller */ struct event_base *base; /* libevent handler created by this thread */ rel_time_t curr_time; /* current time */ struct event clock_event; /* clock event to time each one second */ bool initialized; /* whether clock_event has been initialized */ struct timeval startup_time; /* start time of the thread */ } ms_thread_t; /* initialize threads */ void ms_thread_init(void); /* cleanup some resource of threads when all the threads exit */ void ms_thread_cleanup(void); #ifdef __cplusplus } #endif #endif /* end of MS_THREAD_H */ libmemcached-1.1.4/docs/000077500000000000000000000000001440143131000150315ustar00rootroot00000000000000libmemcached-1.1.4/docs/CMakeLists.txt000066400000000000000000000070311440143131000175720ustar00rootroot00000000000000find_package(Sphinx) if(NOT SPHINX_EXECUTABLE) message(WARNING "The sphinx-build command is required to build documentation.") else() if(NOT DEFINED SPHINX_OPTIONS) set(SPHINX_OPTIONS) endif() if(NOT DEFINED SPHINX_THEME) set(SPHINX_THEME sphinx_rtd_theme) set(SPHINX_THEME_OPTIONS "'collapse_navigation':False, 'navigation_depth':2, 'titles_only':False, 'includehidden':False") endif() set(SPHINX_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/source") # configured documentation tools and intermediate build results set(SPHINX_BUILD_DIR "${CMAKE_CURRENT_BINARY_DIR}") # Sphinx cache with pickled ReST documents set(SPHINX_CACHE_DIR "${SPHINX_BUILD_DIR}/_doctrees") # HTML output directory set(SPHINX_HTML_DIR "${SPHINX_BUILD_DIR}/html") # MAN output directory set(SPHINX_MAN_DIR "${SPHINX_BUILD_DIR}/man") configure_file("conf.py.in" "conf.py" @ONLY) add_subdirectory(source) file(GLOB_RECURSE DOCS_SOURCE CONFIGURE_DEPENDS "*.rst") if(BUILD_DOCS_HTML) add_custom_command( OUTPUT "${SPHINX_HTML_DIR}" COMMAND ${SPHINX_EXECUTABLE} -q -b html -c "${SPHINX_BUILD_DIR}" -d "${SPHINX_CACHE_DIR}" ${SPHINX_OPTIONS} "${SPHINX_SOURCE_DIR}" "${SPHINX_HTML_DIR}" COMMAND ${CMAKE_COMMAND} -E touch "${SPHINX_HTML_DIR}" BYPRODUCTS ${SPHINX_HTML_DIR} DEPENDS "${DOCS_SOURCE}" conf.py.in ) add_custom_target(html ALL DEPENDS "${SPHINX_HTML_DIR}") install(DIRECTORY ${SPHINX_HTML_DIR}/ COMPONENT doc DESTINATION ${CMAKE_INSTALL_DOCDIR}/html) endif() if(BUILD_DOCS_MAN) add_custom_command( OUTPUT ${SPHINX_MAN_DIR} COMMAND ${SPHINX_EXECUTABLE} -q -b man -c "${SPHINX_BUILD_DIR}" -d "${SPHINX_CACHE_DIR}" ${SPHINX_OPTIONS} "${SPHINX_SOURCE_DIR}" "${SPHINX_MAN_DIR}" COMMAND ${CMAKE_COMMAND} -E touch "${SPHINX_MAN_DIR}" BYPRODUCTS ${SPHINX_MAN_DIR} DEPENDS "${DOCS_SOURCE}" conf.py.in ) add_custom_target(man ALL DEPENDS "${SPHINX_MAN_DIR}") set(MAN_EXT "") if(BUILD_DOCS_MANGZ) find_program(PIGZ pigz) if(PIGZ) set(GZIP ${PIGZ}) else() find_package(UnixCommands) endif() if(GZIP) set(MAN_EXT ".gz") add_custom_target(man_gz ALL COMMAND ${GZIP} -kf ${SPHINX_MAN_DIR}/*.1 ${SPHINX_MAN_DIR}/*.3 DEPENDS ${SPHINX_MAN_DIR} ) # some shells do not support braced glob patterns endif() endif() install(DIRECTORY ${SPHINX_MAN_DIR}/ COMPONENT doc DESTINATION ${CMAKE_INSTALL_MANDIR}/man1 FILES_MATCHING PATTERN *.1${MAN_EXT} ) install(DIRECTORY ${SPHINX_MAN_DIR}/ COMPONENT doc DESTINATION ${CMAKE_INSTALL_MANDIR}/man3 FILES_MATCHING PATTERN *.3${MAN_EXT} ) endif() endif() libmemcached-1.1.4/docs/conf.py.in000066400000000000000000000630451440143131000167450ustar00rootroot00000000000000# -*- coding: utf-8 -*- import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ----------------------------------------------------- #needs_sphinx = '1.0' extensions = [@SPHINX_EXTENSIONS@] templates_path = ['_templates'] source_suffix = '.rst' master_doc = 'index' pygments_style = 'sphinx' primary_domain = 'cpp' default_role = 'any' project = u'libmemcached-awesome' version = '@PROJECT_VERSION_MAJOR@.@PROJECT_VERSION_MINOR@' release = '@PROJECT_VERSION@' smartquotes = False m2r_parse_relative_links = True # -- Options for HTML output --------------------------------------------------- html_theme = '@SPHINX_THEME@' html_theme_options = {@SPHINX_THEME_OPTIONS@} html_domain_indices = False html_show_sourcelink = False html_copy_source = False #manpages_url = 'http://man7.org/linux/man-pages/man{section}/{page}.{section}.html' manpages_url = 'https://linux.die.net/man/{section}/{page}' html_context = { 'display_github': True, 'github_user': 'awesomized', 'github_repo': 'libmemcached', 'github_version': 'v1.x/docs/source/' } # -- Options for manual page output -------------------------------------------- # Skip a separate AUTHOR section man_authors = [] # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('libhashkit/index' ,'libhashkit' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_create' ,'hashkit_clone' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_create' ,'hashkit_create' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_create' ,'hashkit_free' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_create' ,'hashkit_is_allocated' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_function' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_get_distribution_function' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_get_function' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_set_custom_distribution_function',u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_set_custom_function' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_set_distribution_function' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_function' ,'hashkit_set_function' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_crc32' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_fnv1_32' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_fnv1_64' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_fnv1a_32' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_fnv1a_64' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_functions' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_hsieh' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_jenkins' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_md5' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_functions' ,'hashkit_murmur' ,u'libhashkit Documentation' ,man_authors,3), ('libhashkit/hashkit_value' ,'hashkit_value' ,u'libhashkit Documentation' ,man_authors,3), ('libmemcached' ,'libmemcached' ,u'C/C++ Client Library for memcached' ,man_authors,3), ('libmemcached/configuration' ,'libmemcached_check_configuration' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/configuration' ,'libmemcached_configuration' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/configuration' ,'memcached' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/examples' ,'libmemcached_examples' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_analyze' ,'memcached_analyze' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_append' ,'memcached_append_by_key' ,u'Appending to or Prepending Data' ,man_authors,3), ('libmemcached/memcached_append' ,'memcached_append' ,u'Appending to or Prepending Data' ,man_authors,3), ('libmemcached/memcached_append' ,'memcached_prepend_by_key' ,u'Appending to or Prepending Data' ,man_authors,3), ('libmemcached/memcached_append' ,'memcached_prepend' ,u'Appending to or Prepending Data' ,man_authors,3), ('libmemcached/memcached_auto' ,'memcached_auto' ,u'Incrementing and Decrementing Values',man_authors,3), ('libmemcached/memcached_auto' ,'memcached_decrement' ,u'Incrementing and Decrementing Values',man_authors,3), ('libmemcached/memcached_auto' ,'memcached_decrement_with_initial' ,u'Incrementing and Decrementing Values',man_authors,3), ('libmemcached/memcached_auto' ,'memcached_increment' ,u'Incrementing and Decrementing Values',man_authors,3), ('libmemcached/memcached_auto' ,'memcached_increment_with_initial' ,u'Incrementing and Decrementing Values',man_authors,3), ('libmemcached/memcached_behavior' ,'memcached_behavior_get' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_behavior' ,'memcached_behavior_set' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_behavior' ,'memcached_behavior' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_callback' ,'memcached_callback_get' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_callback' ,'memcached_callback_set' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_callback' ,'memcached_callback' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_cas' ,'memcached_cas_by_key' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_cas' ,'memcached_cas' ,u'Atomic Compare and Swap' ,man_authors,3), ('libmemcached/memcached_create' ,'memcached_clone' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_create' ,'memcached_create' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_create' ,'memcached_free' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_create' ,'memcached_servers_reset' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_delete' ,'memcached_delete_by_key' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_delete' ,'memcached_delete' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_dump' ,'memcached_dump' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_exist' ,'memcached_exist_by_key' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_exist' ,'memcached_exist' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_fetch' ,'memcached_fetch' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_flush_buffers' ,'memcached_flush_buffers' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_flush' ,'memcached_flush' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_generate_hash_value','memcached_generate_hash' ,u'Generating hash values directly' ,man_authors,3), ('libmemcached/memcached_generate_hash_value','memcached_generate_hash_value' ,u'Generating hash values directly' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_fetch_execute' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_fetch_result' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_get_by_key' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_get' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_mget_by_key' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_mget_execute_by_key' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_mget_execute' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_get' ,'memcached_mget' ,u'Retrieving data from the server' ,man_authors,3), ('libmemcached/memcached_last_error' ,'memcached_last_error_errno' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_last_error' ,'memcached_last_error_message' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_last_error' ,'memcached_last_error' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_memory_allocators' ,'memcached_get_memory_allocators' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_memory_allocators' ,'memcached_memory_allocators' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_memory_allocators' ,'memcached_set_memory_allocators_context' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_memory_allocators' ,'memcached_set_memory_allocators' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_quit' ,'memcached_quit' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_cas' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_create' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_flags' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_free' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_key_length' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_key_value' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_length' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_st' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_result_st' ,'memcached_result_value' ,u'Working with result sets' ,man_authors,3), ('libmemcached/memcached_return_t' ,'memcached_return_t' ,u'Return type values ' ,man_authors,3), ('libmemcached/memcached_sasl' ,'memcached_destroy_sasl_auth_data' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_sasl' ,'memcached_get_sasl_callbacks' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_sasl' ,'memcached_sasl_set_auth_data' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_sasl' ,'memcached_sasl' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_sasl' ,'memcached_set_sasl_callbacks' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_add' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_add_unix_socket' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_count' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_cursor' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_list' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_push' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_server_st' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_servers' ,'memcached_servers' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_server_st' ,'memcached_server_list_append' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_server_st' ,'memcached_server_list_count' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_server_st' ,'memcached_server_list_free' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_server_st' ,'memcached_servers_parse' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_set_encoding_key' ,'memcached_set_encoding_key' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_set' ,'memcached_add_by_key' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_set' ,'memcached_add' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_set' ,'memcached_replace_by_key' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_set' ,'memcached_replace' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_set' ,'memcached_set_by_key' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_set' ,'memcached_set' ,u'Storing and Replacing Data' ,man_authors,3), ('libmemcached/memcached_stats' ,'memcached_stat_execute' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_stats' ,'memcached_stat_get_keys' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_stats' ,'memcached_stat_get_value' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_stats' ,'memcached_stat_servername' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_stats' ,'memcached_stats' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_stats' ,'memcached_stat' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_strerror' ,'memcached_strerror' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_touch' ,'memcached_touch_by_key' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_touch' ,'memcached_touch' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_user_data' ,'memcached_get_user_data' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_user_data' ,'memcached_set_user_data' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_user_data' ,'memcached_user_data' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_verbosity' ,'memcached_verbosity' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_version' ,'memcached_lib_version' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcached/memcached_version' ,'memcached_version' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/index' ,'libmemcachedutil' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_behavior_get' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_behavior_set' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_create' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_destroy' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_fetch' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_pop' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_push' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_release' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool_st' ,u'libmemcached Documentation' ,man_authors,3), ('libmemcachedutil/memcached_pool' ,'memcached_pool' ,u'libmemcached Documentation' ,man_authors,3), ('bin/memcapable' , 'memcapable' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memcat' , 'memcat' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memcp' , 'memcp' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memdump' , 'memdump' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memerror' , 'memerror' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memflush' , 'memflush' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memrm' , 'memrm' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memaslap' , 'memaslap' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memslap' , 'memslap' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memstat' , 'memstat' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memexist' , 'memexist' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memparse' , 'memparse' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memping' , 'memping' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memtouch' , 'memtouch' ,u'libmemcached Documentation' , man_authors, 1), ] if '@CLIENT_PREFIX@' != 'mem' : man_pages.extend([ ('bin/memcapable' , '@CLIENT_PREFIX@capable' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memcat' , '@CLIENT_PREFIX@cat' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memcp' , '@CLIENT_PREFIX@cp' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memdump' , '@CLIENT_PREFIX@dump' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memerror' , '@CLIENT_PREFIX@error' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memflush' , '@CLIENT_PREFIX@flush' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memrm' , '@CLIENT_PREFIX@rm' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memaslap' , '@CLIENT_PREFIX@aslap' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memslap' , '@CLIENT_PREFIX@slap' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memstat' , '@CLIENT_PREFIX@stat' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memexist' , '@CLIENT_PREFIX@exist' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memparse' , '@CLIENT_PREFIX@parse' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memping' , '@CLIENT_PREFIX@ping' ,u'libmemcached Documentation' , man_authors, 1), ('bin/memtouch' , '@CLIENT_PREFIX@touch' ,u'libmemcached Documentation' , man_authors, 1), ]) rst_prolog = """ .. |client_prefix| replace:: @CLIENT_PREFIX@ .. |libhashkit_version| replace:: @LIBHASHKIT_VERSION_INC@ .. |libmemcached_version| replace:: @LIBMEMCACHED_VERSION_INC@ .. |libmemcachedprotocol_version| replace:: @LIBMEMCACHEDPROTOCOL_VERSION_INC@ .. |libmemcachedutil_version| replace:: @LIBMEMCACHEDUTIL_VERSION_INC@ """ @SPHINX_CONF_APPEND@ libmemcached-1.1.4/docs/gh-pages/000077500000000000000000000000001440143131000165245ustar00rootroot00000000000000libmemcached-1.1.4/docs/gh-pages/publish.sh000077500000000000000000000006171440143131000205350ustar00rootroot00000000000000#!/bin/bash set -eu cd "$(dirname $0)" if test -d pages/.git then cd pages git pull -r cd .. else git clone -b gh-pages github.com:awesomized/libmemcached pages fi mkdir -p build cd build cmake -DBUILD_DOCSONLY=true -DBUILD_DOCS_HTML=true ../../.. make html rsync -va --delete --exclude=.git/ docs/html/ ../pages/ cd ../pages touch .nojekyll git add -A git ci -m "update docs" git push libmemcached-1.1.4/docs/source/000077500000000000000000000000001440143131000163315ustar00rootroot00000000000000libmemcached-1.1.4/docs/source/CMakeLists.txt000066400000000000000000000026621440143131000210770ustar00rootroot00000000000000find_program(M2R NAMES m2r m2r-3) function(DefaultRstPath RSTFILE_VAR MDFILE) string(REGEX REPLACE "\\.md$" ".rst" FILEPATH ${MDFILE}) get_filename_component(FILENAME ${FILEPATH} NAME) set(${RSTFILE_VAR} "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}" PARENT_SCOPE) endfunction() function(UpdateRstFromMd RSTFILE MDFILE) if(NOT IS_ABSOLUTE ${MDFILE}) set(MDFILE ${CMAKE_CURRENT_SOURCE_DIR}/${MDFILE}) endif() if (NOT IS_ABSOLUTE ${RSTFILE}) set(RSTFILE ${CMAKE_CURRENT_SOURCE_DIR}/${RSTFILE}) endif() if(${MDFILE} IS_NEWER_THAN ${RSTFILE}) message("-- Processing ${MDFILE} ...") execute_process( COMMAND ${M2R} --overwrite --parse-relative-links ${MDFILE} RESULTS_VARIABLE M2R_STATUS ) if(M2R_STATUS) message(" Failed to update ${RSTFILE}: ${M2R_STATUS}") else() string(REGEX REPLACE "\\.md$" ".rst" FILEPATH ${MDFILE}) file(RENAME ${FILEPATH} ${RSTFILE} ) message(" Updated ${RSTFILE}: OK") endif() endif() endfunction() if(M2R) message("-- Checking ChangeLogs, BUGS, etc...") file(GLOB CHANGELOGS "${CMAKE_SOURCE_DIR}/ChangeLog-*.md") foreach(CHANGELOG IN LISTS CHANGELOGS) DefaultRstPath(RESTFILE ${CHANGELOG}) UpdateRstFromMd(${RESTFILE} ${CHANGELOG}) endforeach() UpdateRstFromMd(issues.rst "${CMAKE_SOURCE_DIR}/BUGS.md") endif() libmemcached-1.1.4/docs/source/ChangeLog-0.rst000066400000000000000000000450431440143131000210550ustar00rootroot00000000000000 ChangeLog v0.x ============== v 0.53 ------ .. released 2011-09-27 * Fix for FreeBSD/OpenBSD and -lm * Added memcached_exist() * Fix for memory when using config test. * CLI gained --quiet v 0.52 ------ .. released 2011-09-12 * Build fixes for Ubuntu/Suse. * Fixes for OSX Lion. * Bug fix for looping back through dns lookups under certain failures. * Fixes related to dead server failures. v 0.51 ------ .. released 2011-07-21 * memcached_callback_set() now takes its data argument as const * Update to tests. * Fix in parser for port number. v 0.50 ------ .. released 2011-06-20 * Updates to C++ interface * Custom free allocators need to now check for value before calling free. * memcached_fetch_result() now uses the internal result when available (about 25 to 50% faster). * Fix for stats structure. * Updates to documentation. * memcached_fetch_result() now uses the internal result when available (about 25 to 50% faster). v 0.49 ------ .. released 2011-04-14 * Fix calls to auto methods so that if value is not passed in nothing bad happens. * New parser calls for generating memcached_st objects. * New error system. * New flow control for messages means faster get/set calls. * Added new documentation system. * A behavior change has been now made that if you specify a weight for any server, we enable the weight flag and do weight balancing. * Added MEMCACHED_BEHAVIOR_REMOVE_FAILED_SERVERS to simplify the setting of AUTO REJECT for servers. v 0.48 ------ .. released 2011-03-16 * Fix memory leak in server parse. * Move test framework out to be its own library (easier to work with Gearman). v 0.47 ------ .. released 2011-02-24 * Additional fixes for OpenBSD. * Bug fix 677609, 456080. * SIGPIPE fix for Linux send(). * memcapable can now test ascii or binary based on flags. * Additional build fixes for SASL. v 0.46 ------ .. released 2011-02-14 * Fixes a number of corner case bugs. * Fixes related to OpenBSD. * Better testing for protocol version. * Removes special case infinite wait on blocking setup. v 0.45 ------ .. released 2011-02-09 * Add support for systemtap v 0.44 ------ .. released 2010-09-23 * Windows bug fixes. * Hudson port support in test harness. * Improved portability of test hanrness. * SASL fixes. v 0.43 ------ .. released 2010-07-28 * Added --args to memstat so that a greater range of values can be returned. * Prelimanary support for Windows. * memcached_stat_execute() merged. v 0.42 ------ .. released 2010-07-06 * Mistake in libtool caused issue with library version v 0.41 ------ .. released 2010-06-30 * Added --file for memcat. * Added limemcached_ping() to libmemcached_util * Bugfix for some cases where connect would have issues with timeout. * Wrong value for errno given as error on an IO failure inside of poll. * Bug fix for issue where multiple interfaces with bad DNS were not being caught. v 0.40 ------ .. released 2010-04-23 * Placed retry logic in for busted resolvers * Add an ignore for SIGPIPE to solve OSX issues. * A couple of fixed for memcached_light server. * Updated to debug mode to track io_wait v 0.39 ------ .. released 2010-04-06 * Add support for prefix keys to binary protocol. * Remove the undocumented call memcached_server_remove(). * The undocumented call memcached_server_by_key() now returns const. * memcached_server_error_reset() has been deprecated. * memcached_server_list() has been deprecated. Use memcached_server_cursor() to walk the servers found in a memcached_st() structure. * memcached_verbosity() can now be run concurrently with other operations. * SASL support. * Fixes memory leak found in EJECT HOSTS. v 0.38 ------ .. released 2010-02-10 * C++ interface for libhashkit. * Modified memcached_set_memory_allocators() so that it requires a context pointer. * memcached_clone() now runs 5 times faster. * Functions used for callbacks are now given const memcached_st. * Added MEMCACHED_BEHAVIOR_CORK. * memslap now creates a configuration file at ~/.memslap.cnf * memcached_purge() now calls any callbacks registered during get execution. * Many fixes to memslap. * Updates for memcapable. * Compile fixes for OpenBSD. * Fix for possible recursive decent on IO failure. v 0.37 ------ .. released 2010-01-12 * Fixed build for libhashkit. * Fixed install path regression. * Modified RPM to strict check install. * Added documentation for memcached_server_cursor(); * Added memcached_servers_reset(). * Modified memcached_st to remove dead cursor_server member. v 0.36 ------ .. released 2010-01-07 * Merged in new memslap utility. * All of constants.h has been updated to match style (all old identifiers continue to work). * Added first pass for libhashkit. * Updated test Framework/extended tests. * Random read support during replication added. * Modified use_sort so that the option can be applied to any distribution type. * We removed the MEMCACHED_BEHAVIOR_KETAMA_COMPAT_MODE added in 0.35. Instead use memcached_behavior_set_distribution(). v 0.35 ------ .. released 2009-11-09 * Added support for by_key operations for inc/dec methods. * Added mget test to memslap. * Support for compatible ketama for SpyMemcached * Update C++ interface. * Fix for memcp v 0.34 ------ .. released 2009-10-13 * Added support for setting behavior flags on a connection pool. * Don't increment server_failure_counter on normal disconnects. * Added prototype for a callback based protocol parser (server side) with examples so that you could let your own application speak the memcached protocol * Updated memcapable to test ASCII protocol. * Changed behavior so that server can be removed at first sign of failure. * Added memcached_server_get_last_disconnect() call v 0.33 ------ .. released 2009-09-23 * Added memcapable to test servers for binary compatibility. * Updated C++ interface. Added basic support for C++ exceptions. Added multiple constructors the memcached client object. The C++ interface now takes parameters which are C++ types (such as std::string). * Several bug fixes for binary protocol support. * Fixed crashing issue with dumping from memcachd server (server internals were changed without documenting change). v 0.32 ------ .. released 2009-09-15 * Change of behavior where linger is only modified for no-block and then it is set to zero. * Added Twitter's memcached_server_error() functions. * Fix for OSX compiles in development builds. * Updated C++ interface. * Updated memcached_mget and memcached_mget_by_key to take a size_t as a parameter instead of an unsigned int for number_of_keys. v 0.31 ------ .. released 2009-07-10 * Added support or HA via replication. * malloc() removed for server key usage. * Update build system. * Added support for memcached_set_memory_allocators(). * Fixed bug in configure.ac for have_htoll. v 0.30 ------ .. released 2009-06-01 * Added memcachd_dump command (and framework for memdump tool). * Realigned all structures to remove padding (and line up important bits for 64bit caches. * Remove some of sprintf() in storage calls(). * Removed printf() in stat call for unknown stat member. * memcached_generate_hash() function added. * Added tests to make sure all hash functions are stable. v 0.29 ------ .. released 2009-05-19 * Fixed malloc usage to calloc for spots where we need zero filled memory. * All code warnings now treated as errors. * Fixes for debian packaging. * Added new pooling mechanism. * MEMCACHED_BEHAVIOR_NO_BLOCK no longer also sets MEMCACHED_BEHAVIOR_BUFFER_REQUESTS. * Updated generic rpm. v 0.28 ------ .. released 2009-04-15 * Fixed bug in init sructure (reapplied) * Fixed bug in get/set by key (nikkhils@gmail.com) v 0.27 ------ .. released 2009-03-30 * Added new UDP fire-forget mode. * Reworked performance for mget() to better make use of async protocol * Cleaned up execution of fetch (just one set of code now) * Fixed Jenkin's for big endian hosts. * Updates for memstat to determine network latency. * Updates for binary protocol. * Many updates to documentation. v 0.26 ------ .. released 2009-01-29 * Fix for decrement on hash key * Fixed assert that was catching bad memset() call in host_reset() * Fix purge issue for blocked IO which has been stacked. v 0.25 ------ .. released 2008-11-28 * Jenkins HASH added. * Update of Murmur hash code * Support explicit weights (Robey Pointer, Evan Weaver) * Bugfix for ketama continuum (Robey Pointer) * New behavior MEMCACHED_BEHAVIOR_HASH_WITH_PREFIX_KEY (Robey Pointer) * Don't ever call stats for weighting servers, because it is unstable. v 0.24 ------ .. released 2008-09-16 * Cleanup compile warnings. * Fix issues in partitioning by keys. * Fixed "fail case" to make sure when calling memcached_clone() no memcached_st is over written. * New memcached_server_by_key() method for finding a server from a key. * memcached_server_free() was added for freeing server structures. v 0.23 ------ .. released 2008-09-07 * Added strings.h header for Solaris 9 * Solaris 64bit fix. * Support for weighted Ketama from Yin Chen. * Fix for Chinese * Fix for 0 length key to trigger bad key. * Added behaviors MEMCACHED_BEHAVIOR_SND_TIMEOUT, MEMCACHED_BEHAVIOR_RCV_TIMEOUT * Support for Binary Protocol added v 0.22 ------ .. released 2008-07-14 * Fix where master key was no being checked for "bad key" * Fixed bugs in stats output (thread output was wrong) * Clarified MEMCACHED_BAD_KEY_PROVIDED is return for bad prefix key. * Found a bug in Flags return (Jacek Ostrowski) * Fixed issue with compiling on Visual Studio v 0.21 ------ .. released 2008-05-24 * Change of char * to const char * for all key based functions. * New MEMCACHED_CALLBACK_PREFIX_KEY added. You can now create domains for values. * Fixed bug introducd in last version on memcp * Fix for death of file io to call shutdown() v 0.20 ------ .. released 2008-05-05 * New consistent distribution tests. * Found a memory leak when a server constantly fails. * Fix in watchpoint macro * Changed default timeout to 1 second for poll timeouts * Wheel uses less memory/dynamic allocation for size (no longer limited to 512 hosts by default. * memslap memory leak fix * Added Ketama distribution * Fix assert.h compile problem on CentOS v 0.19 ------ .. released 2008-04-09 * Documentation fix in libmemcached. * Fixed bug where sort was always occuring on hosts * Logic fix in branch prediction (thanks Jay!) * Read through cached support. * Fixed for cas by key operation. * Fix for memcached_server_st list structures to have correct count. * Added callback MEMCACHED_CALLBACK_DELETE_TRIGGER * Removed function call in favor of macro (aka cut out some instructions) v 0.18 ------ .. released 2008-03-17 * Fix plus tests for non-zero value objects and flags. * MEMCACHED_HASH_MURMUR added for murmur algorithm provided. * MEMCACHED_BEHAVIOR_RETRY_TIMEOUT added to keep connecting from looping on timeout. * gcc branch prediction optimizations * Refactored entire tree to make include files cleaner * Fixed leaked socket. v 0.17 ------ .. released 2008-02-27 * MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT added for connect timeout in non-block mode. * Incompatible change in memcached_behavior_set() api. We now use a uint64_t, instead of a pointer. * Fix for storage of values for zero. * memcached_server_cursor() function added to API for cycling through servers. v 0.16 ------ .. released 2008-02-18 * Work on the UDP protocol * Added get_by_key, set_by_key tests for C++ API * Fix for limit_maxbytes to be 64bit in stats * Added Atom Smasher test (scale baby, scale!) * Servers are now sorted, meaning that servers are now ordered so that clients with the same lists, will have same distribution. (Idea from Ross McFarland). MEMCACHED_BEHAVIOR_SORT_HOSTS was added to enable this support. * Added MEMCACHED_BAD_KEY_PROVIDED error for auto, set, and get operations. MEMCACHED_BEHAVIOR_VERIFY_KEY was added to enable this feature. * More error messages on command line tools. * Fixed bugs in memcached_cas() operator. * Fix to loop through interfaces v 0.15 ------ .. released 2008-01-29 * More work on the C++ API. * Bug fixes around block corner cases. * Slight performance increase in both read() and write(). v 0.14 ------ .. released 2008-01-22 * For for bug found by Evan Weaver where increment() was not returning propper error of value was not found. * Fix for bad null pointer on flag by Toru Maesaka. * Refactor of all IO to just pass in the active server * Problem configuring (PKG_CHECK_MODULES) fixed by removal of "rpath" in support/libmemcached.pc.in (Thanks to Ross McFarland). * Added memcached_callback_get()/set() * First prototype of C++ interface * Updated docs for uint16_t changes in previous release v 0.13 ------ .. released 2008-01-13 * MEMCACHED_BEHAVIOR_USER_DATA added to store user pointer. * Fix for failure to connect to invalidate socket. * Patch from Marc Rossi to add --hash option for memcp, memrm, and memcat. * Kevin's patch for fixing EOF issues during a read. * Toru Maesaka patch for stats mismatch * Fix for when CRC return 0 * Fixed uint16_t issues around flags. Turns out the documentation on the protocol was wrong. * Lingering socket fixes for FreeBSD. * Patches from Kevin Dalley for FreeBSD 4.0 * Added multi delete functions. * All get key returns have C style null termination * If memcached_server_list_append is passed NULLs instead of pointers it returns NULL. * Added memcached_fetch_execute() method * Found a bug where memcached_fetch() was not null terminating the result value. * memcached_behavior() now has the ability to set "buffering" so that data is not automatically flushed. * Behavior change, buffered commands now return MEMCACHED_BUFFERED v 0.12 ------ .. released 2007-12-11 * Updates for consistent hashing * IPV6 support * Static allocation for hostname (performance) * Fixed bug where in non-block mode all data might not have been sent on close(). * Refactor of memcached_get() to use common code. * Change in value fetch, MEMCACHED_END is now returned when keys are no longer in the pipe. * Fixed bug where key could be out of range of characters * Added _by_key() methods to allow partitioning of values to particular servers. * MEMCACHED_DEFAILT_TIMEOUT is now set to a non -1 value. * Performance improvements in get operations. v 0.11 ------ .. released 2007-11-26 * Added option to memcache_behavior_set() so that poll() can be timed out. * Fixed memory leak in case of using memcached_fetch_result() where no value was returned. * Bug fixed in memcached_connect() which would cause servers that did not need to be enabled to be enabled (performance issue). * Rewrote bounds checking code for get calls. * "make test" now starts its own memcached servers. * Added Hseih hash (MEMCACHED_HASH_HSIEH), which is showing about 7% performance over standard hash. v 0.10 ------ .. released 2007-11-21 * Added append binary test. * Added MEMCACHED_BEHAVIOR_CACHE_LOOKUPS behavior so that you can save on multiple DNS lookups. * Added CAS support, though this is optional and must be enabled during runtime. * Added the utility memerror to create human readable error strings from memcached errors (aka convert ints to strings) * Fixed type in MEMCACHED_HOST_LOOKUP_FAILURE * Fixed bug where hostname might not be null terminated * Moved to using gethostbyname_r() on Linux to solve thread safety issue * Added -rpath support for pkg-config * Documentation fix for hash setting using memcached_behavior_set() v 0.9 ----- .. released 2007-11-15 * fix for when no servers are definied. * different buffers are now kept for different connections to speed up async efforts * Modified increment/decrement functions to return uint64_t values * Fixed bug in cases where zero length keys were provided * Thread cleanup issue in memslap * No hostname lookup on reconnect * Fix for flag settings (was doing hex by accident!) * Support for 1.2.4 server additions "prepend" and "append" added. * Added memcached_version()... not sure if I will make this public or not. v 0.8 ----- .. released 2007-11-05 * Adding support for CRC hash method * Adding support for UNIX sockets * Added additional HASHing methods of FNV1_64,FNV1A_64, FNV1_32, FNV1A_32 * Added pkgconfig support (PKG_CHECK_MODULES) * Fixed conflict with defined type in MySQL * Added memcached_result_st structure and functions to manipulate it. v 0.7 ----- .. released 2007-10-30 * Poved to poll() from select() * Fixes in internal string class for allocation of large numbers of strings. * memcached_mget() function now sends keys as it parses them instead of building strings as it goes. * Propper flush now for making sure we get all IO sent even when in non-block mode. * Added --enable-debug rule for configure * All asserts() removed (hey this is going into production!) v 0.6 ----- .. released 2007-10-17 * get value returns are now null terminated (request by Cal Heldenbrand) * Fixed connections for more hosts then two. * Rewrite of the read/write IO systems to handle different sorts of host failures. * Added man pages for all functions and tools * Raised buffer size for readinng/writing to 16K * You can now optionally set the socket size for recv/send via memached_behavior_set/get. v 0.5 ----- .. released 2007-10-09 * Ruby maintainer mentioned TCP_NODELAY patch he had added. Added this to C library as well. (Eric Hodel drbrain@segment7.net) * Added support script for set_benchmark * Updated memslap to allow testing of TCP_NODELAY * Updated memslap to support --flush (aka dump memcache servers before testing) * Fixed bug in multiple hosts not being activated * Added environmental variable MEMCACHED_SERVERS which can be used to set the servers list. * fixed memcached_stat method (and now memstat works) * server connect now happens on demand. * Help for all command line applications v 0.4 ----- .. released 2007-10-03 * Added buffered IO to write calls for keys * Added buffered IO for reads * memstat was broken (bad if/else on connect) * New non-blocking IO (not default yet). Mucho faster * Refactor of test system. * memslap crash solved v 0.3 ----- .. released 2007-10-01 * Jeff Fisher guppy@techmonkeys.org provided a spec file * Added "make rpm" around dist file * Added support for Solaris * Added support for DTrace * Fixed read to be recv and write to be send * Bug fix where memstat would core if no server was found * Added memslap tool (load generator) * Numerous bug fixes in library * Added calls to library for creating host lists (see text cases to understand how to use this). v 0.2 ----- .. released 2007-09-27 * First public version libmemcached-1.1.4/docs/source/ChangeLog-1.0.rst000066400000000000000000000063051440143131000212120ustar00rootroot00000000000000 ChangeLog v1.0 ============== v 1.0.18 -------- .. released 2014-02-09 * MEMCACHED_BEHAVIOR_RETRY_TIMEOUT can now be set to zero. * Numerous bug fixes. v 1.0.17 -------- .. released 2013-04-03 * Remove c++ namespace that was being exposed (the API should be plug compatible).. * Fix cases where --servers wasn't behaving the same in all clients. v 1.0.16 -------- .. released 2013-02-01 * Added support to do two part shutdown of socket. * Fixes for Fedora 18. * Fix for binary memcached_touch() v 1.0.15 -------- .. released 2012-12-17 * Added support for Murmur3 (HASHKIT_HASH_MURMUR3) * Portability fixes. v 1.0.14 -------- .. released 2012-11-14 * CLIENT_ERROR fixed to not be treated as a fatal error. * Compiler fixes for older Ubuntu releases. v 1.0.13 -------- .. released 2012-10-19 * Fix bug that caused version string to not be exported correctly. v 1.0.12 -------- .. released 2012-10-09 * Added memcached_result_take_value(). * Added ax_libmemcached.m4 v 1.0.11 -------- .. released 2012-09-17 * Removed custom version of memcached. * Updated hardening rules. * Fixed a case where the return error from a socket connection differred from that of a TCP/IP socket. v 1.0.10 -------- .. released 2012-07-30 * --disable-assert has been removed from configure, and --enable-assert has been added in its place. * Compiling fixes for Clang on OSX Mountain Lion. v 1.0.9 ------- .. released 2012-07-05 * Faster close on socket. * Instance allocation is now seperated from server interface. This should allow for a better preservation of ABI compliance from now on. * Fix close on exec bug. * Numerous other bug fixes. v 1.0.8 ------- .. released 2012-05-22 * Added support for setting options via ENV variable LIBMEMCACHED * Fix corner case on last used result. v 1.0.7 ------- .. released 2012-04-28 * Add API call for exist calls. * Update all license files to be BSD. v 1.0.6 ------- .. released 2012-04-08 * Fixes for gcc 4.7, lp:961812 * Fix for restart issue that happens under testing. * Fix for lp:962815. * Support for transparent AES encryption. v 1.0.5 ------- .. released 2012-03-14 * Fixes for OSX. * Version is now parsed directly in the parser, which makes buffered operations now work with it.. * memstat has been extended so that it can be used to find the version of the server. * Update documentation. * Fixes for compile issues on Debian and Ubuntu v 1.0.4 ------- .. released 2012-01-27 * Fix for memcached_dump(). * Additional testing for memcached_stat_execute(). v 1.0.3 ------- .. released 2012-01-09 * Increased size of sort buffer used during Ketama. * Added support for new behavior to handle dead servers. * Overall haul of UDP IO. * Fixed C compile issue with memcached_exist() * Numerous bug fixes. * Clang support for OSX. * All commands now using vector send support. v 1.0.2 ------- .. released 2011-10-24 * Dropped libmemcached/memcached_util.h (undocumented header file) * Added memcached_touch() and memcached_touch_by_key() * UDP support restructured to toggle on a complete memcached_st structure. ---- See :doc:`ChangeLog-0 <./ChangeLog-0>` for changes prior v1.0. libmemcached-1.1.4/docs/source/ChangeLog-1.1.rst000066400000000000000000000241161440143131000212130ustar00rootroot00000000000000.. role:: raw-html-m2r(raw) :format: html ChangeLog v1.1 ============== v 1.1.4 ------- .. released 2022-03-06 * Fix `gh #107 `_\ : macOS: deprecated sasl API (improve detection of ``libsasl2``\ ). * Fix `gh #131 `_\ : Consider renaming tools (add ``CLIENT_PREFIX`` build option; default: ``mem``\ ) * Fix `gh #132 `_\ : Add build of static library (add ``BUILD_SHARED_LIBS`` build option; default: ``ON``\ ). * Fix `gh #134 `_\ : Update client option documentation. * Fix `gh #136 `_\ : ``libmemcachedutil`` is underlinked (link against libmemcached). * Fix `gh php-memcached#531 `_\ : ``get`` returns random values when lower than default ``OPT_POLL_TIMEOUT`` is set. v 1.1.3 ------- .. released 2022-11-09 * Fix `gh #130 `_ with `gh #124 `_\ : Server response count can underflow. v 1.1.2 ------- .. released 2022-08-10 * Fix handling of negative expiration values, which are somehow allowed by legacy.\ :raw-html-m2r:`
` See also `gh #125 `_\ , and `gh #76 `_. * Fix `gh #122 `_\ : If libcrypto implementation of AES is used, do not compile internal. * Fix missing include of :raw-html-m2r:`` in tests. * Fix warnings with non-SASL builds. * Fix pthread.h detection. v 1.1.1 ------- .. released 2021-09-16 * Fix `gh #67 `_\ : GET returns ``NOTFOUND`` on ``TIMEOUT``. * Fix `gh #113 `_\ : Build failure with Catch2 < 2.13.5. * Add `gh #114 `_\ : Add possibility to use libcrypto for encryption. * Add `gh #115 `_\ : Add ``LIBMEMCACHED_AWESOME`` CPP define. * Add test for `gh #75 `_\ : memcached_clone of SASL connection closes random file descriptor. * Fix `gh #116 `_\ : Add libmemcachedpotocol-0-0/configure.h guarding ``ssize_t`` typedef. * Fix `gh #120 `_\ : libmemcached.pc is missing a ``Requires`` entry for libsasl2. v 1.1.0 ------- .. released 2021-06-23 **Changes from beta3:** * Add ASCII multi get support to bin/memslap. See logs from ``beta3``\ , ``beta2``\ , and ``beta1`` for the full list of changes since the last 1.0 release. v 1.1.0-beta3 ------------- .. released 2021-04-15 **Changes from beta2:** * Fix `gh #108 `_\ : macOS Big Sur: dtrace does not understand -G switch. * Add support for IPv6 bracketed syntax in ``memcached_servers_parse``. * Make ``memcat``\ 's ``--file`` option's argument optional defaulting to ````. * Fix libmemcachedprotocol's binary ``STAT`` and ``VERSION`` handlers. * Fix `gh #105 `_\ : EINTR handled too defensively when polling. v 1.1.0-beta2 ------------- .. released 2020-12-28 **Changes from beta1:** * Fix `gh #103 `_\ : Build failure on 32-bit. * Fix `gh #102 `_\ : Doc build with old sphinx. * Fix `gh #100 `_\ : Revert symbolic rename of public header include directories. * Fix `gh #98 `_\ : Library SONAMEs and NAME_LINKs differ from 1.0.18. * Fix `gh #97 `_\ : Location of cmake files installation directory. * Fix `gh #96 `_\ : LIBXXX_VERSION_HEX constants format. v 1.1.0-beta1 ------------- .. released 2020-12-21 **NOTE:**\ :raw-html-m2r:`
` This is a bug fix release, not a feature release. The minor version number was incremented due to the following changes: * Ported build system to CMake. * Ported test suite to Catch2. * Build requires C++11 compiler support. * Tests require C++17 compiler support. * Moved to the Semantic Versioning Specification: https://semver.org * Moved the project from launchpad to github: * Source: https://github.com/awesomized/libmemcached * Documentation: https://awesomized.github.io/libmemcached * Continuous Integration: * Github: https://github.com/awesomized/libmemcached/actions (Linux, MacOS, Windows **·** amd64) * Sourcehut: https://builds.sr.ht/~m6w6/libmemcached (FreeBSD, OpenBSD **·** amd64) * Build artifacts: https://artifacts.m6w6.name/libmemcached/ rsync://m6w6.name::artifacts/libmemcached/ * Fix build failure due to comparison of incompatible types in bin/memflush and bin/memstat. * Fix wrong type of memcached_instance_st::server_timeout_counter_query_id from uint32_t to uint64_t. * Fix memcached_dump(): returned MEMCACHED_CLIENT_ERROR on request to dump illegal slab id. * Fix bin/memcapable: failed with "No hostname was provided" when providing a hostname. * Fix hashkit/murmur and hashkit/murur3: undefined behavior on platforms requiring aligned access. * Fix Memcache::set(): possible subscription of empty vector. * Fix libmemcached_util_version_check(). * Fix ketama/consistent hashing: crash on reallocation of continuum. * Fix `gh #90 `_\ : Build fails on Darwin. * Fix `gh #83 `_\ : memcp waits forever if file no found. * Fix `gh #80 `_\ : memparse docs. * Fix `gh #72 `_ and `gh #47 `_\ : memcached_return_t docs. * Fix `gh #62 `_\ : uint32_t overflow cause busy loop. * Removed restriction of UDP+IPv6. * Fix SERVER_ERROR_MEMORY_ALLOCATION_FAILURE: recognize more strings returned by the server. * Fix `gh #13 `_\ : reset continuum counter after freeing them. * Fix `gh #14 `_ and `gh #17 `_\ : SASL: AUTH_CONTINUE was considered a failure and caused IO reset. * Fix `gh #25 `_\ : hashkit/murmur3 unavailable. * Fix missing handling of EAGAIN for non-blocking unix domain socket. * Fix `gh #35 `_\ : handling of BEHAVIOR_REMOVE_FAILED_SERVERS. * Fix `gh #41 `_\ : ensure stable sort on continuum host key collision. * Fix `gh #42 `_\ : MEMCACHED_MAX_BUFFER docs. * Fix `gh #43 `_\ : libmemcached_configuration docs. * Fix `gh #46 `_\ : clarification on millisecond timeout docs. * Fix `gh #50 `_\ : memcached_fetch_result() can return previously returned data. * Fix `gh #53 `_\ : stack overflow in memcached_fetch_result(). * Fix `gh #57 `_\ : include vs :raw-html-m2r:`` * Fix `gh #58 `_\ : more specific error messages when connect() fails. * Fix `gh #59 `_\ : bin/memcat: typo in "No servers provied". * Fix `gh #77 `_\ : undeclared UINT64_C in ketama.cc. * Fix `gh #12 `_\ : never reconnects after connection reset (binary protocol). * Fix `gh #49 `_\ : assertion memcached_failed(rc) failed in memcached_send_ascii(). * Fix `gh #67 `_\ : get returns NOTFOUND on timeout. * Fix `gh #76 `_\ : memcached_touch() crashes when expiration=-1 (ASCII only). * Fix `gh #23 `_\ : build fails with bison 2.3. * Fix memaslap: build fails with newer compiler versions. * Fix usage of strerror_r() implementations returning pointer to char. * Fix pipelining commands with memcached >= 1.6. * Fix memcached_stat_get_value(): buffer overflow. * Fix memcached_stat(): undefined behavior due to unintialized memcached_return_t. * Fix SASL tests: requires SASL_PWDB_CONF. * Fix bin/memaslap to idnentify itself as memaslap instead of memslap. * Fix bin/memcapable to work with memcached >= 1.6. * Fix murmur and murmur3 hashes on big endian platforms. * Fix `gh #82 `_\ , `gh #64 `_ and `gh #21 `_\ : clarify documentation on replication. * Fix `gh #95 `_\ : MEMCACHED_CALLBACK_GET_FAILURE and MEMCACHED_BEHAVIOR_BUFFER_REQUESTS * Fix bin/memcat to output flags if requested with ``--flag``. * Fix `gh #68 `_\ : Windows support. ---- See :doc:`ChangeLog-1.0 <./ChangeLog-1.0>` for changes prior v1.1. libmemcached-1.1.4/docs/source/bin/000077500000000000000000000000001440143131000171015ustar00rootroot00000000000000libmemcached-1.1.4/docs/source/bin/common/000077500000000000000000000000001440143131000203715ustar00rootroot00000000000000libmemcached-1.1.4/docs/source/bin/common/env.rst000066400000000000000000000001301440143131000217050ustar00rootroot00000000000000ENVIRONMENT ----------- .. envvar:: MEMCACHED_SERVERS Specify a list of servers. libmemcached-1.1.4/docs/source/bin/common/note_contrib_options.rst000066400000000000000000000002111440143131000253550ustar00rootroot00000000000000 CONTRIBUTED PROGRAM ................... This is a contributed program. This program doesn't follow the standard flag/option scheme. libmemcached-1.1.4/docs/source/bin/common/note_program_prefix.rst000066400000000000000000000006631440143131000252010ustar00rootroot00000000000000 PROGRAM PREFIX .............. The prefix of this program is variable, i.e. it can be configured at build time. Usually the client programs of ``libmemcached-awesome`` are prefixed with ``mem``, like ``memcat`` or ``memcp``. It can be configured, though, to replace the prefix with something else like ``mc``, in case of that, the client programs of ``libmemcached-awesome`` would be called ``mccat``, ``mccp``, etc. respectively. libmemcached-1.1.4/docs/source/bin/index.rst000066400000000000000000000020141440143131000207370ustar00rootroot00000000000000Client Applications =================== Data Manipulation ----------------- .. toctree:: :titlesonly: memexist — Check for the existence of a key memcat — "cat" data from a server memcp — "cp" files to a server memtouch — "touch" a key memrm – "rm" a key Tests and Analysis ------------------ .. toctree:: :titlesonly: memaslap - Load testing and benchmarking a server memslap - Load testing and benchmarking a server memping – Ping a server memstat – Gather statistics from a server memerror — Translate libmemcached error codes memparse — Parse and validate an option string memcapable — Check a server's capabilities and compatibility Server Administration --------------------- .. toctree:: :titlesonly: memdump — Dump a server's data memflush — Flush a server (erase all cached data) libmemcached-1.1.4/docs/source/bin/memaslap.rst000066400000000000000000000720151440143131000214370ustar00rootroot00000000000000================================================== memaslap - Load testing and benchmarking a server ================================================== SYNOPSIS -------- |client_prefix|\aslap [options] .. program:: memaslap .. option:: --help .. option:: --servers .. envvar:: MEMCACHED_SERVERS DESCRIPTION ----------- :program:`memaslap` is a load generation and benchmark tool for memcached servers. It generates configurable workload such as threads, concurrency, connections, run time, overwrite, miss rate, key size, value size, get/set proportion, expected throughput, and so on. Furthermore, it also tests data verification, expire-time verification, UDP, binary protocol, facebook test, replication test, multi-get and reconnection, etc. Memaslap manages network connections like memcached with libevent. Each thread of memaslap is bound with a CPU core, all the threads don't communicate with each other, and there are several socket connections in each thread. Each connection keeps key size distribution, value size distribution, and command distribution by itself. You can specify servers via the :option:`memaslap --servers` option or via the environment variable :envvar:`MEMCACHED_SERVERS`. FEATURES -------- Memslap is developed to for the following purposes: Manages network connections with libevent asynchronously. Set both TCP and UDP up to use non-blocking IO. Improves parallelism: higher performance in multi-threads environments. Improves time efficiency: faster processing speed. Generates key and value more efficiently; key size distribution and value size distribution are configurable. Supports get, multi-get, and set commands; command distribution is configurable. Supports controllable miss rate and overwrite rate. Supports data and expire-time verification. Supports dumping statistic information periodically. Supports thousands of TCP connections. Supports binary protocol. Supports facebook test (set with TCP and multi-get with UDP) and replication test. DETAILS ------- Effective implementation of network. ____________________________________ For memaslap, both TCP and UDP use non-blocking network IO. All the network events are managed by libevent as memcached. The network module of memaslap is similar to memcached. Libevent can ensure memaslap can handle network very efficiently. Effective implementation of multi-threads and concurrency _________________________________________________________ Memslap has the similar implementation of multi-threads to memcached. Memslap creates one or more self-governed threads; each thread is bound with one CPU core if the system tests setting CPU core affinity. In addition, each thread has a libevent to manage the events of the network; each thread has one or more self-governed concurrency; and each concurrency has one or more socket connections. All the concurrent tasks don't communicate with each other even though they are in the same thread. Memslap can create thousands of socket connections, and each concurrency has tens of socket connections. Each concurrency randomly or sequentially selects one socket connection from its socket connection pool to run, so memaslap can ensure each concurrency handles one socket connection at any given time. Users can specify the number of concurrency and socket connections of each concurrency according to their expected workload. Effective implementation of generating key and value ____________________________________________________ In order to improve time efficiency and space efficiency, memaslap creates a random characters table with 10M characters. All the suffixes of keys and values are generated from this random characters table. Memslap uses the offset in the character table and the length of the string to identify a string. It can save much memory. Each key contains two parts, a prefix and a suffix. The prefix is an uint64_t, 8 bytes. In order to verify the data set before, memaslap need to ensure each key is unique, so it uses the prefix to identify a key. The prefix cannot include illegal characters, such as '\r', '\n', '\0' and ' '. And memaslap has an algorithm to ensure that. Memslap doesn't generate all the objects (key-value pairs) at the beginning. It only generates enough objects to fill the task window (default 10K objects) of each concurrency. Each object has the following basic information, key prefix, key suffix offset in the character table, key length, value offset in the character table, and value length. In the work process, each concurrency sequentially or randomly selects an object from the window to do set operation or get operation. At the same time, each concurrency kicks objects out of its window and adds new object into it. Simple but useful task scheduling _________________________________ Memslap uses libevent to schedule all concurrent tasks of threads, and each concurrency schedules tasks based on the local task window. Memslap assumes that if each concurrency keeps the same key distribution, value distribution and commands distribution, from outside, memaslap keeps all the distribution as a whole. Each task window includes a lot of objects, each object stores its basic information, such as key, value, expire time, and so on. At any time, all the objects in the window keep the same and fixed key and value distribution. If an object is overwritten, the value of the object will be updated. Memslap verifies the data or expire-time according to the object information stored in the task window. Libevent selects which concurrency to handle based on a specific network event. Then the concurrency selects which command (get or set) to operate based on the command distribution. If it needs to kick out an old object and add a new object, in order to keep the same key and value distribution, the new object must have the same key length and value length. If memcached server has two cache layers (memory and SSD), running memaslap with different window sizes can get different cache miss rates. If memaslap adds enough objects into the windows at the beginning, and the cache of memcached cannot store all the objects initialized, then memaslap will get some objects from the second cache layer. It causes the first cache layer to miss. So the user can specify the window size to get the expected miss rate of the first cache layer. Useful implementation of multi-servers , UDP, TCP, multi-get and binary protocol ________________________________________________________________________________ Because each thread is self-governed, memaslap can assign different threads to handle different memcached servers. This is just one of the ways in which memaslap tests multiple servers. The only limitation is that the number of servers cannot be greater than the number of threads. The other way to test multiple servers is for replication test. Each concurrency has one socket connection to each memcached server. For the implementation, memaslap can set some objects to one memcached server, and get these objects from the other servers. By default, Memslap does single get. If the user specifies multi-get option, memaslap will collect enough get commands and pack and send the commands together. Memslap tests both the ASCII protocol and binary protocol, but it runs on the ASCII protocol by default. Memslap by default runs on the TCP protocol, but it also tests UDP. Because UDP is unreliable, dropped packages and out-of-order packages may occur. Memslap creates a memory buffer to handle these problems. Memslap tries to read all the response data of one command from the server and reorders the response data. If some packages get lost, the waiting timeout mechanism can ensure half-baked packages will be discarded and the next command will be sent. USAGE ----- Below are some usage samples: memaslap -s 127.0.0.1:11211 -S 5s memaslap -s 127.0.0.1:11211 -t 2m -v 0.2 -e 0.05 -b memaslap -s 127.0.0.1:11211 -F config -t 2m -w 40k -S 20s -o 0.2 memaslap -s 127.0.0.1:11211 -F config -t 2m -T 4 -c 128 -d 20 -P 40k memaslap -s 127.0.0.1:11211 -F config -t 2m -d 50 -a -n 40 memaslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m memaslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m -p 2 The user must specify one server at least to run memaslap. The rest of the parameters have default values, as shown below: Thread number = 1 Concurrency = 16 Run time = 600 seconds Configuration file = NULL Key size = 64 Value size = 1024 Get/set = 9:1 Window size = 10k Execute number = 0 Single get = true Multi-get = false Number of sockets of each concurrency = 1 Reconnect = false Data verification = false Expire-time verification = false ASCII protocol = true Binary protocol = false Dumping statistic information periodically = false Overwrite proportion = 0% UDP = false TCP = true Limit throughput = false Facebook test = false Replication test = false Key size, value size and command distribution. ______________________________________________ All the distributions are read from the configuration file specified by user with "—cfg_cmd" option. If the user does not specify a configuration file, memaslap will run with the default distribution (key size = 64, value size = 1024, get/set = 9:1). For information on how to edit the configuration file, refer to the "Configuration File" section. The minimum key size is 16 bytes; the maximum key size is 250 bytes. The precision of proportion is 0.001. The proportion of distribution will be rounded to 3 decimal places. The minimum value size is 1 bytes; the maximum value size is 1M bytes. The precision of proportion is 0.001. The proportion of distribution will be rounded to 3 decimal places. Currently, memaslap only tests set and get commands. And it testss 100% set and 100% get. For 100% get, it will preset some objects to the server. Multi-thread and concurrency ____________________________ The high performance of memaslap benefits from the special schedule of thread and concurrency. It's important to specify the proper number of them. The default number of threads is 1; the default number of concurrency is 16. The user can use "—threads" and "--concurrency" to specify these variables. If the system tests setting CPU affinity and the number of threads specified by the user is greater than 1, memaslap will try to bind each thread to a different CPU core. So if you want to get the best performance memaslap, it is better to specify the number of thread equal to the number of CPU cores. The number of threads specified by the user can also be less or greater than the number of CPU cores. Because of the limitation of implementation, the number of concurrencies could be the multiple of the number of threads. 1. For 8 CPU cores system For example: --threads=2 --concurrency=128 --threads=8 --concurrency=128 --threads=8 --concurrency=256 --threads=12 --concurrency=144 2. For 16 CPU cores system For example: --threads=8 --concurrency=128 --threads=16 --concurrency=256 --threads=16 --concurrency=512 --threads=24 --concurrency=288 The memaslap performs very well, when used to test the performance of memcached servers. Most of the time, the bottleneck is the network or the server. If for some reason the user wants to limit the performance of memaslap, there are two ways to do this: Decrease the number of threads and concurrencies. Use the option "--tps" that memaslap provides to limit the throughput. This option allows the user to get the expected throughput. For example, assume that the maximum throughput is 50 kops/s for a specific configuration, you can specify the throughput equal to or less than the maximum throughput using "--tps" option. Window size ___________ Most of the time, the user does not need to specify the window size. The default window size is 10k. For Schooner Memcached, the user can specify different window sizes to get different cache miss rates based on the test case. Memslap testss cache miss rate between 0% and 100%. If you use this utility to test the performance of Schooner Memcached, you can specify a proper window size to get the expected cache miss rate. The formula for calculating window size is as follows: Assume that the key size is 128 bytes, and the value size is 2048 bytes, and concurrency=128. 1. Small cache cache_size=1M, 100% cache miss (all data get from SSD). win_size=10k 2. cache_size=4G (1). cache miss rate 0% win_size=8k (2). cache miss rate 5% win_size=11k 3. cache_size=16G (1). cache miss rate 0% win_size=32k (2). cache miss rate 5% win_size=46k The formula for calculating window size for cache miss rate 0%: cache_size / concurrency / (key_size + value_size) \* 0.5 The formula for calculating window size for cache miss rate 5%: cache_size / concurrency / (key_size + value_size) \* 0.7 Verification ____________ Memslap testss both data verification and expire-time verification. The user can use "--verify=" or "-v" to specify the proportion of data verification. In theory, it testss 100% data verification. The user can use "--exp_verify=" or "-e" to specify the proportion of expire-time verification. In theory, it testss 100% expire-time verification. Specify the "--verbose" options to get more detailed error information. For example: --exp_verify=0.01 –verify=0.1 , it means that 1% of the objects set with expire-time, 10% of the objects gotten will be verified. If the objects are gotten, memaslap will verify the expire-time and value. multi-servers and multi-config _______________________________ Memslap testss multi-servers based on self-governed thread. There is a limitation that the number of servers cannot be greater than the number of threads. Memslap assigns one thread to handle one server at least. The user can use the "--servers=" or "-s" option to specify multi-servers. For example: --servers=10.1.1.1:11211,10.1.1.2:11212,10.1.1.3:11213 --threads=6 --concurrency=36 The above command means that there are 6 threads, with each thread having 6 concurrencies and that threads 0 and 3 handle server 0 (10.1.1.1); threads 1 and 4 handle server 1 (10.1.1.2); and thread 2 and 5 handle server 2 (10.1.1.3). All the threads and concurrencies in memaslap are self-governed. So is memaslap. The user can start up several memaslap instances. The user can run memaslap on different client machines to communicate with the same memcached server at the same. It is recommended that the user start different memaslap on different machines using the same configuration. Run with execute number mode or time mode _________________________________________ The default memaslap runs with time mode. The default run time is 10 minutes. If it times out, memaslap will exit. Do not specify both execute number mode and time mode at the same time; just specify one instead. For example: --time=30s (It means the test will run 30 seconds.) --execute_number=100000 (It means that after running 100000 commands, the test will exit.) Dump statistic information periodically. ________________________________________ The user can use "--stat_freq=" or "-S" to specify the frequency. For example: --stat_freq=20s Memslap will dump the statistics of the commands (get and set) at the frequency of every 20 seconds. For more information on the format of dumping statistic information, refer to "Format of Output" section. Multi-get _________ The user can use "--division=" or "-d" to specify multi-get keys count. Memslap by default does single get with TCP. Memslap also testss data verification and expire-time verification for multi-get. Memslap testss multi-get with both TCP and UDP. Because of the different implementation of the ASCII protocol and binary protocol, there are some differences between the two. For the ASCII protocol, memaslap sends one "multi-get" to the server once. For the binary protocol, memaslap sends several single get commands together as "multi-get" to the server. UDP and TCP ___________ Memslap testss both UDP and TCP. For TCP, memaslap does not reconnect the memcached server if socket connections are lost. If all the socket connections are lost or memcached server crashes, memaslap will exit. If the user specifies the "--reconnect" option when socket connections are lost, it will reconnect them. User can use "--udp" to enable the UDP feature, but UDP comes with some limitations: UDP cannot set data more than 1400 bytes. UDP is not tested by the binary protocol because the binary protocol of memcached does not tests that. UDP doesn't tests reconnection. Facebook test _____________ Set data with TCP and multi-get with UDP. Specify the following options: "--facebook --division=50" If you want to create thousands of TCP connections, specify the "--conn_sock=" option. For example: --facebook --division=50 --conn_sock=200 The above command means that memaslap will do facebook test, each concurrency has 200 socket TCP connections and one UDP socket. Memslap sets objects with the TCP socket, and multi-gets 50 objects once with the UDP socket. If you specify "--division=50", the key size must be less that 25 bytes because the UDP packet size is 1400 bytes. Replication test ________________ For replication test, the user must specify at least two memcached servers. The user can use "—rep_write=" option to enable feature. For example: --servers=10.1.1.1:11211,10.1.1.2:11212 –rep_write=2 The above command means that there are 2 replication memcached servers, memaslap will set objects to both server 0 and server 1, get objects which are set to server 0 before from server 1, and also get objects which are set to server 1 before from server 0. If server 0 crashes, memaslap will only get objects from server 1. If server 0 comes back to life again, memaslap will reconnect server 0. If both server 0 and server 1 crash, memaslap will exit. Supports thousands of TCP connections _____________________________________ Start memaslap with "--conn_sock=" or "-n" to enable this feature. Make sure that your system can tests opening thousands of files and creating thousands of sockets. However, this feature does not tests reconnection if sockets disconnect. For example: --threads=8 --concurrency=128 --conn_sock=128 The above command means that memaslap starts up 8 threads, each thread has 16 concurrencies, each concurrency has 128 TCP socket connections, and the total number of TCP socket connections is 128 \* 128 = 16384. Supports binary protocol ________________________ Start memaslap with "--binary" or "-B" options to enable this feature. It testss all the above features except UDP, because the latest memcached 1.3.3 does not implement binary UDP protocol. For example: --binary Since memcached 1.3.3 doesn't implement binary UDP protocol, memaslap does not tests UDP. In addition, memcached 1.3.3 does not tests multi-get. If you specify "--division=50" option, it just sends 50 get commands together as "multi-get" to the server. Configuration file ------------------ This section describes the format of the configuration file. By default when no configuration file is specified memaslap reads the default one located at ~/.memaslap.cnf. Below is a sample configuration file: .. code-block:: perl --------------------------------------------------------------------------- #comments should start with '#' #key #start_len end_len proportion # #key length range from start_len to end_len #start_len must be equal to or greater than 16 #end_len must be equal to or less than 250 #start_len must be equal to or greater than end_len #memaslap will generate keys according to the key range #proportion: indicates keys generated from one range accounts for the total generated keys # #example1: key range 16~100 accounts for 80% # key range 101~200 accounts for 10% # key range 201~250 accounts for 10% # total should be 1 (0.8+0.1+0.1 = 1) # # 16 100 0.8 # 101 200 0.1 # 201 249 0.1 # #example2: all keys length are 128 bytes # # 128 128 1 key 128 128 1 #value #start_len end_len proportion # #value length range from start_len to end_len #start_len must be equal to or greater than 1 #end_len must be equal to or less than 1M #start_len must be equal to or greater than end_len #memaslap will generate values according to the value range #proportion: indicates values generated from one range accounts for the total generated values # #example1: value range 1~1000 accounts for 80% # value range 1001~10000 accounts for 10% # value range 10001~100000 accounts for 10% # total should be 1 (0.8+0.1+0.1 = 1) # # 1 1000 0.8 # 1001 10000 0.1 # 10001 100000 0.1 # #example2: all value length are 128 bytes # # 128 128 1 value 2048 2048 1 #cmd #cmd_type cmd_proportion # #currently memaslap only testss get and set command. # #cmd_type #set 0 #get 1 # #example: set command accounts for 50% # get command accounts for 50% # total should be 1 (0.5+0.5 = 1) # # cmd # 0 0.5 # 1 0.5 cmd 0 0.1 1.0 0.9 Format of output ---------------- At the beginning, memaslap displays some configuration information as follows: servers : 127.0.0.1:11211 threads count: 1 concurrency: 16 run time: 20s windows size: 10k set proportion: set_prop=0.10 get proportion: get_prop=0.90 Where _____ servers : "servers" The servers used by memaslap. threads count The number of threads memaslap runs with. concurrency The number of concurrencies memaslap runs with. run time How long to run memaslap. windows size The task window size of each concurrency. set proportion The proportion of set command. get proportion The proportion of get command. The output of dynamic statistics is something like this: .. code-block:: perl --------------------------------------------------------------------------------------------------------------------------------- Get Statistics Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) Avg(us) Std_dev Geo_dist Period 5 345826 69165 65.3 0 27 2198 203 95.43 177.29 Global 20 1257935 62896 71.8 0 26 3791 224 117.79 192.60 Set Statistics Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) Avg(us) Std_dev Geo_dist Period 5 38425 7685 7.3 0 42 628 240 88.05 220.21 Global 20 139780 6989 8.0 0 37 3790 253 117.93 224.83 Total Statistics Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) Avg(us) Std_dev Geo_dist Period 5 384252 76850 72.5 0 27 2198 207 94.72 181.18 Global 20 1397720 69886 79.7 0 26 3791 227 117.93 195.60 --------------------------------------------------------------------------------------------------------------------------------- Where _____ Get Statistics Statistics information of get command Set Statistics Statistics information of set command Total Statistics Statistics information of both get and set command Period Result within a period Global Accumulated results Ops Total operations TPS Throughput, operations/second Net The rate of network Get_miss How many objects can't be gotten Min The minimum response time Max The maximum response time Avg: The average response time Std_dev Standard deviation of response time Geo_dist Geometric distribution based on natural exponential function At the end, memaslap will output something like this: .. code-block:: perl --------------------------------------------------------------------------------------------------------------------------------- Get Statistics (1257956 events) Min: 26 Max: 3791 Avg: 224 Geo: 192.60 Std: 116.23 Log2 Dist: 4: 0 10 84490 215345 8: 484890 459823 12543 824 12: 31 Set Statistics (139782 events) Min: 37 Max: 3790 Avg: 253 Geo: 224.84 Std: 116.83 Log2 Dist: 4: 0 0 4200 16988 8: 50784 65574 2064 167 12: 5 Total Statistics (1397738 events) Min: 26 Max: 3791 Avg: 227 Geo: 195.60 Std: 116.60 Log2 Dist: 4: 0 10 88690 232333 8: 535674 525397 14607 991 12: 36 cmd_get: 1257969 cmd_set: 139785 get_misses: 0 verify_misses: 0 verify_failed: 0 expired_get: 0 unexpired_unget: 0 written_bytes: 242516030 read_bytes: 1003702556 object_bytes: 152086080 packet_disorder: 0 packet_drop: 0 udp_timeout: 0 Run time: 20.0s Ops: 1397754 TPS: 69817 Net_rate: 59.4M/s --------------------------------------------------------------------------------------------------------------------------------- Where _____ Get Statistics Get statistics of response time Set Statistics Set statistics of response time Total Statistics Both get and set statistics of response time Min The accumulated and minimum response time Max The accumulated and maximum response time Avg The accumulated and average response time Std Standard deviation of response time Log2 Dist Geometric distribution based on logarithm 2 cmd_get Total get commands done cmd_set Total set commands done get_misses How many objects can't be gotten from server verify_misses How many objects need to verify but can't get them verify_failed How many objects with insistent value expired_get How many objects are expired but we get them unexpired_unget How many objects are unexpired but we can't get them written_bytes Total written bytes read_bytes Total read bytes object_bytes Total object bytes packet_disorder How many UDP packages are disorder packet_drop How many UDP packages are lost udp_timeout How many times UDP time out happen Run time Total run time Ops Total operations TPS Throughput, operations/second Net_rate The average rate of network OPTIONS ------- -s, --servers= List one or more servers to connect. Servers count must be less than threads count. e.g.: --servers=localhost:1234,localhost:11211 -T, --threads= Number of threads to startup, better equal to CPU numbers. Default 8. -c, --concurrency= Number of concurrency to simulate with load. Default 128. -n, --conn_sock= Number of TCP socks per concurrency. Default 1. -x, --execute_number= Number of operations(get and set) to execute for the given test. Default 1000000. -t, --time= How long the test to run, suffix: s-seconds, m-minutes, h-hours, d-days e.g.: --time=2h. -F, --cfg_cmd= Load the configure file to get command,key and value distribution list. -w, --win_size= Task window size of each concurrency, suffix: K, M e.g.: --win_size=10k. Default 10k. -X, --fixed_size= Fixed length of value. -v, --verify= The proportion of date verification, e.g.: --verify=0.01 -d, --division= Number of keys to multi-get once. Default 1, means single get. -S, --stat_freq= Frequency of dumping statistic information. suffix: s-seconds, m-minutes, e.g.: --resp_freq=10s. -e, --exp_verify= The proportion of objects with expire time, e.g.: --exp_verify=0.01. Default no object with expire time -o, --overwrite= The proportion of objects need overwrite, e.g.: --overwrite=0.01. Default never overwrite object. -R, --reconnect Reconnect tests, when connection is closed it will be reconnected. -U, --udp UDP tests, default memaslap uses TCP, TCP port and UDP port of server must be same. -a, --facebook Whether it enables facebook test feature, set with TCP and multi-get with UDP. -B, --binary Whether it enables binary protocol. Default with ASCII protocol. -P, --tps= Expected throughput, suffix: K, e.g.: --tps=10k. -p, --rep_write= The first nth servers can write data, e.g.: --rep_write=2. -b, --verbose Whether it outputs detailed information when verification fails. -h, --help Display this message and then exit. -V, --version Display the version of the application and then exit. EXAMPLES -------- memaslap -s 127.0.0.1:11211 -S 5s memaslap -s 127.0.0.1:11211 -t 2m -v 0.2 -e 0.05 -b memaslap -s 127.0.0.1:11211 -F config -t 2m -w 40k -S 20s -o 0.2 memaslap -s 127.0.0.1:11211 -F config -t 2m -T 4 -c 128 -d 20 -P 40k memaslap -s 127.0.0.1:11211 -F config -t 2m -d 50 -a -n 40 memaslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m memaslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m -p 2 NOTES ----- .. include:: common/note_program_prefix.rst .. include:: common/note_contrib_options.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` libmemcached-1.1.4/docs/source/bin/memcapable.rst000066400000000000000000000020451440143131000217220ustar00rootroot00000000000000memcapable ========== SYNOPSIS -------- .. program:: memcapable |client_prefix|\capable [options] Check a memcached server's capabilities and compatibility. DESCRIPTION ----------- :program:`memcapable` connects to the specified memcached server and tries to determine its capabilities by running various commands and verifying the response. OPTIONS ------- .. option:: -h hostname Specify the hostname to connect to. The default is \ *localhost*\ . .. option:: -p port Specify the port number to connect to. The default is \ *11211*\ . .. option:: -c :manpage:`abort(3)` when detecting an error from the server. .. option:: -v Print out the comparison when it detects an error from the server. .. option:: -t n Set the timeout for an IO operation to/from the server to \ *n*\ seconds. NOTES ----- .. include:: common/note_program_prefix.rst .. include:: common/note_contrib_options.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` .. only:: html * :doc:`/libmemcached` libmemcached-1.1.4/docs/source/bin/memcat.rst000066400000000000000000000015571440143131000211110ustar00rootroot00000000000000memcat ====== SYNOPSIS -------- .. program:: memcat |client_prefix|\cat [options] key [key...] Read and output the value of one key or the values of a set of keys. DESCRIPTION ----------- :program:`memcat` reads and outputs the value of a single or a set of keys stored in a :manpage:`memcached(1)` server. If any key is not found an error is returned. It is similar to the standard UNIX :manpage:`cat(1)` utility. OPTIONS ------- .. include:: options/common_get.rst .. include:: options/flags_noarg.rst .. include:: options/hash.rst .. include:: options/file_out.rst .. include:: common/env.rst NOTES ----- .. include:: common/note_program_prefix.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` :manpage:`libmemcached_configuration(3)` .. only:: html * :doc:`/libmemcached` * :doc:`/libmemcached/configuration` libmemcached-1.1.4/docs/source/bin/memcp.rst000066400000000000000000000025641440143131000207430ustar00rootroot00000000000000memcp ===== SYNOPSIS -------- .. program:: memcp |client_prefix|\cp [options] \-\-servers Copy files to a collection of memcached servers. DESCRIPTION ----------- :program:`memcp` copies one or more files into :manpage:`memcached(1)` servers. It is similar to the standard UNIX :manpage:`cp(1)` command. The key names will be the names of the files, without any directory path. OPTIONS ------- .. include:: options/common_set.rst .. include:: /bin/options/flags_reqarg.rst .. include:: options/udp.rst .. option:: -S|--set Issue *SET* command(s). This is the default mode. See also :option:`-A|--add` and :option:`-R|--replace`. .. option:: -A|--add Issue *ADD* command(s). .. option:: -R|--replace Issue *REPLACE* command(s). .. option:: -.|--basename Use basename of path as key (default). .. option:: -+|--relative Use relative path (as passed), instead of basename only. .. option:: -/|--absolute Use absolute path (real path), instead of basename only. .. include:: common/env.rst NOTES ----- .. include:: common/note_program_prefix.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` :manpage:`memcached_behavior(3)` .. only:: html * :doc:`/libmemcached` * :doc:`/libmemcached/configuration` * :doc:`/libmemcached/memcached_behavior` libmemcached-1.1.4/docs/source/bin/memdump.rst000066400000000000000000000012741440143131000213030ustar00rootroot00000000000000memdump ======= SYNOPSIS -------- .. program:: memdump |client_prefix|\dump [options] Dump a list of keys from a server. DESCRIPTION ----------- :program:`memdump` dumps a list of "keys" from all servers that it is told to fetch from. Because memcached does not guarantee to provide all keys it is not possible to get a complete "dump". OPTIONS ------- .. include:: options/common_get.rst .. include:: options/file_out.rst .. include:: common/env.rst NOTES ----- .. include:: common/note_program_prefix.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` .. only:: html * :doc:`/libmemcached` * :doc:`/libmemcached/memcached_dump` libmemcached-1.1.4/docs/source/bin/memerror.rst000066400000000000000000000011551440143131000214650ustar00rootroot00000000000000memerror ======== SYNOPSIS -------- .. program:: memerror |client_prefix|\error [options] Translate a memcached error code into a string. DESCRIPTION ----------- :program:`memerror` translates an error code from `libmemcached` into a human readable string. OPTIONS ------- .. include:: options/all.rst .. include:: options/common.rst .. include:: common/env.rst NOTES ----- .. include:: common/note_program_prefix.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` .. only:: html * :doc:`/libmemcached` * :doc:`/libmemcached/index_errors` libmemcached-1.1.4/docs/source/bin/memexist.rst000066400000000000000000000011101440143131000214570ustar00rootroot00000000000000memexist ======== SYNOPSIS -------- .. program:: memexist |client_prefix|\exist [options] Check for the existence of a key. DESCRIPTION ----------- :program:`memexist` checks for the existence of a key within a cluster. OPTIONS ------- .. include:: options/common_get.rst .. include:: options/hash.rst .. include:: common/env.rst NOTES ----- .. include:: common/note_program_prefix.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` .. only:: html * :doc:`/libmemcached` * :doc:`/libmemcached/memcached_exist` libmemcached-1.1.4/docs/source/bin/memflush.rst000066400000000000000000000036531440143131000214620ustar00rootroot00000000000000memflush ======== SYNOPSIS -------- |client_prefix|\flush [options] .. program:: memflush Reset a server or list of servers DESCRIPTION ----------- :program:`memflush` resets the contents of :manpage:`memcached(1)` servers. .. warning:: This means that all data in the specified servers will be deleted. OPTIONS ------- .. include:: options/common_get.rst .. include:: options/expire.rst .. note:: Using an expiration time (period), all keys, which have not bean updated until expiration will cease to exist. Quoting the `memcached protocol documentation`_, it states: Its effect is to invalidate all existing items immediately (by default) or after the expiration specified. After invalidation none of the items will be returned in response to a retrieval command (unless it's stored again under the same key *after* flush_all has invalidated the items). The most precise definition of what flush_all does is the following: it causes all items whose update time is earlier than the time at which flush_all was set to be executed to be ignored for retrieval purposes. The intent of flush_all with a delay, was that in a setting where you have a pool of memcached servers, and you need to flush all content, you have the option of not resetting all memcached servers at the same time (which could e.g. cause a spike in database load with all clients suddenly needing to recreate content that would otherwise have been found in the memcached daemon). .. _memcached protocol documentation: https://github.com/memcached/memcached/blob/master/doc/protocol.txt .. include:: common/env.rst NOTES ----- .. include:: common/note_program_prefix.rst SEE ALSO -------- .. only:: man :manpage:`memcached(1)` :manpage:`libmemcached(3)` .. only:: html * :doc:`/libmemcached` * :doc:`/libmemcached/memcached_flush` libmemcached-1.1.4/docs/source/bin/memparse.rst000066400000000000000000000010261440143131000214430ustar00rootroot00000000000000memparse ======== SYNOPSIS -------- .. program:: memparse |client_prefix|\parse