You are viewing a single comment's thread from:

RE: Run even larger AI models locally with LM Studio

in LeoFinance • last year

I'm getting 23 tokens per second using the 5 bit Mixtal 2.7 model.

last year in LeoFinance by
100%
    -100%

      Downvoting a post can decrease pending rewards and make it less visible. Common reasons:

      • Disagreement on rewards
      • Fraud or plagiarism
      • Hate speech or trolling
      • Miscategorized content or spam

      Submit
      0.00 ARCHON
        4 votes
        Reply 1
        Sort:  
      • Trending
        • [-]
         last year 

        macs have a big edge for this.
        I would recommend the 4 bit, the 5 bit isn't much better and takes a lot more ram. I'd stick with 4 bit, or something like 8 bit if you can get there.

        100%
          -100%

            Downvoting a post can decrease pending rewards and make it less visible. Common reasons:

            • Disagreement on rewards
            • Fraud or plagiarism
            • Hate speech or trolling
            • Miscategorized content or spam

            Submit
            0.00 ARCHON
              4 votes
              Reply